Getting Started

Welcome! If you’re ready to get started with Meltano and run an EL(T) pipeline with a data source and destination of your choosing, you’ve come to the right place!

Install Meltano #

Before you can get started with Meltano and the meltano CLI, you’ll need to install it onto your system.

To learn more about the different installation methods, refer to the Installation guide.

Local installation #

If you’re running Linux, macOS, or Windows and have Python 3.7, 3.8 or 3.9 installed, we recommend installing Meltano into a dedicated Python virtual environment inside the directory that will hold your Meltano projects.

1. Create and navigate to a directory to hold your Meltano projects:

    mkdir meltano-projects
    cd meltano-projects
   

2. Install the pipx package manager:

    python3 -m install --user pipx
    python3 -m pipx ensurepath
    #Note that the below commands are not needed in most cases
    source ~/.bashrc 
   

   For Windows, instead of source ~/.bashrc, you'll want to open a new PowerShell instance.

3. Install the meltano package from PyPI:

    pipx install meltano
   

4. Optionally, verify that the meltano CLI is now available by viewing the version:

    meltano --version
   

If anything’s not behaving as expected, refer to the “Local Installation” section of the Installation guide for more details.

Docker installation #

Alternatively, and assuming you already have Docker installed and running, you can use the meltano/meltano Docker image which exposes the meltano CLI command as its entrypoint.

1. Pull or update the latest version of the Meltano Docker image:

    docker pull meltano/meltano:latest
   

   By default, this image comes with the oldest version of Python supported by Meltano, currently Python 3.8. If you’d like to use a newer version of Python instead, add a -python<X.Y> suffix to the image tag, e.g. latest-python3.9.

2. Optionally, verify that the meltano CLI is now available through the Docker image by viewing the version:

    docker run meltano/meltano --version
   

   Now, whenever this guide or the documentation asks you to run the meltano command, you’ll need to run it using docker run meltano/meltano <args> as in the example above.

   When running a meltano subcommand that requires access to your project (which you’ll create in the next step), you’ll also need to mount the project directory into the container and set it as the container’s working directory:

    docker run -v $(pwd):/project -w /project meltano/meltano <args>
   

   If anything’s not behaving as expected, refer to the “Installing on Docker” section of the Installation guide for more details.

Create your Meltano project #

Now that you have a way of running the meltano CLI, it’s time to create a new Meltano project that (among other things) will hold the plugins that implement the various details of your ELT pipelines.

To learn more about Meltano projects, refer to the Projects concept doc.

1. Navigate to the directory that you’d like to hold your Meltano projects, if you didn’t already do so earlier:

    mkdir meltano-projects
    cd meltano-projects
   

2. Initialize a new project in a directory of your choosing using meltano init:

    meltano init <project directory name>
   
    # For example:
    meltano init my-meltano-project
   
    # If you're using Docker, don't forget to mount the current working directory:
    docker run -v $(pwd):/projects -w /projects meltano/meltano init my-meltano-project
   

   This will create a new directory with, among other things, your meltano.yml project file:

    version: 1
    project_id: <random UUID>
   

   It doesn’t define any plugins, environments, or pipeline schedules yet. Note that anonymous usage stats are enabled by default, if you’re curious and want to learn more about how the product benefits from them or how to change the default settings see the settings reference page for more details.

3. Navigate to the newly created project directory:

    cd <project directory>
   
    # For example:
    cd my-meltano-project
   

4. Optionally, if you’d like to version control your changes, initialize a Git repository and create an initial commit:

    git init
    git add --all
    git commit -m 'Initial Meltano project'
   

   This will allow you to use git diff to easily check the impact of the meltano commands you’ll run below on your project files, most notably your meltano.yml project file.

View and activate your environments #

As part of creating your Meltano project, we automatically added your first environments called dev, staging and prod. This allows you to define configurations specific to the environment you’re running your project in.

1. List your available environments:

    meltano environment list
   

2. Activate your environment for your shell session:

    export MELTANO_ENVIRONMENT=dev
   

   Alternatively you can include the --environment=dev argument to each meltano command. You should now see a log message that says Environment 'dev' is active each time you run a meltano command.

3. [optional] Add a new environment:

    meltano environment add <environment name>
   

Add an extractor to pull data from a source #

Now that you have your very own Meltano project, it’s time to add some plugins to it!

The first plugin you’ll want to add is an extractor, which will be responsible for pulling data out of your data source.

To learn more about adding plugins to your project, refer to the Plugin Management guide.

1. Find out if an extractor for your data source is supported out of the box by checking the Extractors list or using meltano discover:

    meltano discover extractors
   

2. Depending on the result, pick your next step:

   • If an extractor is supported out of the box, add it to your project using meltano add:

    meltano add extractor <plugin name>
   
    # For example:
    meltano add extractor tap-gitlab
   
    # If you have a preference for a non-default variant, select it using `--variant`:
    meltano add extractor tap-gitlab --variant=singer-io
   
    # If you're using Docker, don't forget to mount the project directory:
    docker run -v $(pwd):/project -w /project meltano/meltano add extractor tap-gitlab
   

   This will add the new plugin to your meltano.yml project file:

    plugins:
    extractors:
    - name: tap-gitlab
      variant: meltanolabs
      pip_url: git+https://github.com/MeltanoLabs/tap-gitlab.git
   

   You can now continue to step 4.

   • If an extractor is not yet discoverable, find out if a Singer tap for your data source already exists by checking out MeltanoHub for Singer, which is the best place to find and explore existing Singer taps and targets.

3. Depending on the result, pick your next step:

   • If a Singer tap for your data source is available, add it to your project as a custom plugin using meltano add --custom:

       meltano add --custom extractor <tap name>
     
       # For example:
       meltano add --custom extractor tap-covid-19
     
       # If you're using Docker, don't forget to mount the project directory,
       # and ensure that interactive mode is enabled so that Meltano can ask you
       # additional questions about the plugin and get your answers over STDIN:
       docker run --interactive -v $(pwd):/project -w /project meltano/meltano add --custom extractor tap-covid-19
     

     Meltano will now ask you some additional questions to learn more about the plugin.

     This will add the new plugin to your meltano.yml project file:

       plugins:
         extractors:
         - name: tap-covid-19
           namespace: tap_covid_19
           pip_url: tap-covid-19
           executable: tap-covid-19
           capabilities:
           - catalog
           - discover
           - state
           settings:
           - name: api_token
           - name: user_agent
           - name: start_date
     

     To learn more about adding custom plugins, refer to the Plugin Management guide.

     Once you've got the extractor working in your project, please consider contributing its description to the index of discoverable plugins so that it can be supported out of the box for new users!

   • If a Singer tap for your data source doesn’t exist yet, learn how to build and use your own tap by following the “Create and Use a Custom Extractor” tutorial.

   Once you’ve got your new tap project set up, you can add it to your Meltano project as a custom plugin by following the meltano add --custom instructions above. When asked to provide a pip install argument, you can provide a local directory path or Git repository URL.

4. Optionally, verify that the extractor was installed successfully and that its executable can be invoked using meltano invoke:

    meltano invoke <plugin> --help
   
    # For example:
    meltano invoke tap-gitlab --help
   

   If you see the extractor’s help message printed, the plugin was definitely installed successfully, but an error message related to missing configuration or an unimplemented --help flag would also confirm that Meltano can invoke the plugin’s executable.

Configure the extractor #

Chances are that the extractor you just added to your project will require some amount of configuration before it can start extracting data.

To learn more about managing the configuration of your plugins, refer to the Configuration guide.

extractors:
- name: tap-example
  config: {
    "setting": "value",
    "another_setting": true
  }

1. Find out what settings your extractor supports using meltano config <plugin> list:

    meltano config <plugin> list
   
    # For example:
    meltano config tap-gitlab list
   

2. Assuming the previous command listed at least one setting, set appropriate values using meltano config <plugin> set:

   See MeltanoHub for details on how to get a GitLab `private_token` for tap-gitlab.

    meltano config <plugin> set <setting> <value>
   
    # For example:
    meltano config tap-gitlab set projects "meltano/meltano meltano/tap-gitlab"
    meltano config tap-gitlab set start_date 2021-03-01T00:00:00Z
    meltano config tap-gitlab set private_token my_private_token
   

   This will add the non-sensitive configuration to your meltano.yml project file:

    environments:
    - name: dev
      config:
        plugins:
          extractors:
          - name: tap-gitlab
            config:
              projects: meltano/meltano meltano/tap-gitlab
              start_date: '2021-10-01T00:00:00Z'
   

   Sensitive configuration (like private_token) will instead be stored in your project’s .env file so that it will not be checked into version control:

    export TAP_GITLAB_PRIVATE_TOKEN=my_private_token
   

3. Optionally, verify that the configuration looks like what the Singer tap expects according to its documentation using meltano config <plugin>:

    meltano config <plugin>
   
    # For example:
    meltano config tap-gitlab
   

   This will show the current configuration:

    {
      "api_url": "https://gitlab.com",
      "private_token": "my_private_token",
      "groups": "",
      "projects": "meltano/meltano meltano/tap-gitlab",
      "ultimate_license": false,
      "fetch_merge_request_commits": false,
      "fetch_pipelines_extended": false,
      "start_date": "2021-03-01T00:00:00Z"
    }
   

Select entities and attributes to extract #

Now that the extractor has been configured, it’ll know where and how to find your data, but not yet which specific entities and attributes (tables and columns) you’re interested in.

By default, Meltano will instruct extractors to extract all supported entities and attributes, but it’s recommended that you specify the specific entities and attributes you’d like to extract, to improve performance and save on bandwidth and storage.

To learn more about selecting entities and attributes for extraction, refer to the Data Integration (EL) guide.

1. Find out whether the extractor supports entity selection, and if so, what entities and attributes are available, using meltano select --list --all:

    meltano select <plugin> --list --all
   
    # For example:
    meltano select tap-gitlab --list --all
   

   If this command fails with an error, this usually means that the Singer tap does not support catalog discovery mode, and will always extract all supported entities and attributes.

2. Assuming the previous command succeeded, select the desired entities and attributes for extraction using meltano select:

    meltano select <plugin> <entity> <attribute>
    meltano select <plugin> --exclude <entity> <attribute>
   
    # For example:
    meltano select tap-gitlab commits id
    meltano select tap-gitlab commits project_id
    meltano select tap-gitlab commits created_at
    meltano select tap-gitlab commits author_name
    meltano select tap-gitlab commits message
   
    # Include all attributes of an entity
    meltano select tap-gitlab tags "*"
   
    # Exclude matching attributes of all entities
    meltano select tap-gitlab --exclude "*" "*_url"
   

   As you can see in the example, entity and attribute identifiers can contain wildcards (*) to match multiple entities or attributes at once.

   This will add the selection rules to your meltano.yml project file:

    plugins:
      extractors:
      - name: tap-gitlab
        variant: meltanolabs
        pip_url: git+https://github.com/MeltanoLabs/tap-gitlab.git
    environments:
    - name: dev
      config:
        plugins:
          extractors:
          - name: tap-gitlab
            config:
              projects: meltano/meltano meltano/tap-gitlab
              start_date: '2021-03-01T00:00:00Z'
            select:
            - commits.id
            - commits.project_id
            - commits.created_at
            - commits.author_name
            - commits.message
            - tags.*
            - '!*.*_url'
   

   Note that exclusion takes precedence over inclusion. If an attribute is excluded, there is no way to include it back without removing the exclusion pattern first. This is also detailed in the CLI documentation for the --exclude parameter.

3. Optionally, verify that only the intended entities and attributes are now selected using meltano select --list:

    meltano select <plugin> --list
   
    # For example:
    meltano select tap-gitlab --list
   

Choose how to replicate each entity #

If the data source you’ll be pulling data from is a database, like PostgreSQL or MongoDB, your extractor likely requires one final setup step: setting a replication method for each selected entity (table).

Most database extractors, on the other hand, support two or more of the following replication methods and require you to choose an appropriate option for each table through the replication-method stream metadata key:

• LOG_BASED: Log-based Incremental Replication

  The extractor uses the database’s binary log files to identify what records were inserted, updated, and deleted from the table since the last run (if any), and extracts only these records.

  This option is not supported by all databases and database extractors.

• INCREMENTAL: Key-based Incremental Replication

  The extractor uses the value of a specific column on the table (the Replication Key, e.g. an updated_at timestamp or incrementing id integer) to identify what records were inserted or updated (but not deleted) since the last run (if any), and extracts only those records.

• FULL_TABLE: Full Table Replication

  The extractor extracts all available records in the table on every run.

  To learn more about replication methods, refer to the Data Integration (EL) guide.

  1. Find out which replication methods (i.e. options for the replication-method stream metadata key) the extractor supports by checking its documentation or the README in its repository.

  2. Set the desired replication-method metadata for each selected entity using meltano config <plugin> set and the extractor’s metadata extra:

      meltano config <plugin> set _metadata <entity> replication-method <LOG_BASED|INCREMENTAL|FULL_TABLE>
     
      # For example:
      meltano config tap-postgres set _metadata some_entity_id replication-method INCREMENTAL
      meltano config tap-postgres set _metadata other_entity replication-method FULL_TABLE
     
      # Set replication-method metadata for all entities
      meltano config tap-postgres set _metadata '*' replication-method INCREMENTAL
     
      # Set replication-method metadata for matching entities
      meltano config tap-postgres set _metadata '*_full' replication-method FULL_TABLE
     

     As you can see in the example, entity identifiers can contain wildcards (*) to match multiple entities at once.

     If you’ve set a table’s replication-method to INCREMENTAL, also choose a Replication Key by setting the replication-key metadata:

      meltano config <plugin> set _metadata <entity> replication-key <column>
     
      # For example:
      meltano config tap-postgres set _metadata some_entity_id replication-key updated_at
      meltano config tap-postgres set _metadata some_entity_id replication-key id
     

     This will add the metadata rules to your meltano.yml project file:

      environments:
      - name: dev
        config:
          plugins:
            extractors:
            - name: tap-gitlab
              metadata:
                some_entity_id:
                  replication-method: INCREMENTAL
                  replication-key: id
                other_entity:
                  replication-method: FULL_TABLE
                '*':
                  replication-method: INCREMENTAL
                '*_full':
                  replication-method: FULL_TABLE
     

  3. Optionally, verify that the stream metadata for each table was set correctly in the extractor’s generated catalog file by dumping it using meltano invoke --dump=catalog <plugin>:

      meltano invoke --dump=catalog <plugin>
     
      # For example:
      meltano invoke --dump=catalog tap-postgres
     

Add a loader to send data to a destination #

Now that your Meltano project has everything it needs to pull data from your source, it’s time to tell it where that data should go!

This is where the loader comes in, which will be responsible for loading extracted data into an arbitrary data destination.

To learn more about adding plugins to your project, refer to the Plugin Management guide.

1. Find out if a loader for your data destination is supported out of the box by checking the Loaders list or using meltano discover:

    meltano discover loaders
   

2. Depending on the result, pick your next step:

   • If a loader is supported out of the box, add it to your project using meltano add:

       meltano add loader <plugin name>
     
       # For this example, we'll use the default variant:
       meltano add loader target-postgres
     
       # Or if you just want to use a non-default variant you can use this,
       # selected using `--variant`:
       meltano add loader target-postgres --variant=datamill-co
     

     Sometimes extractors and loaders expect that certain dependencies are already installed. If you run into any issues while installing, refer to MeltanoHub for more help troubleshooting or join the Meltano Slack workspace to ask questions.

     This will add the new plugin to your meltano.yml project file:

       plugins:
       loaders:
       - name: target-postgres
         variant: transferwise
         pip_url: pipelinewise-target-postgres
     

     You can now continue to step 4.

   • If a loader is not yet discoverable, find out if a Singer target for your data source already exists by checking Singer’s index of targets and/or doing a web search for Singer target <data destination>, e.g. Singer target BigQuery.

3. Depending on the result, pick your next step:

   • If a Singer target for your data destination is available, add it to your project as a custom plugin using meltano add --custom:

       meltano add --custom loader <target name>
     
       # For example:
       meltano add --custom loader target-bigquery
     
       # If you're using Docker, don't forget to mount the project directory,
       # and ensure that interactive mode is enabled so that Meltano can ask you
       # additional questions about the plugin and get your answers over STDIN:
       docker run --interactive -v $(pwd):/project -w /project meltano/meltano add --custom loader target-bigquery
     

     Meltano will now ask you some additional questions to learn more about the plugin.

     This will add the new plugin to your meltano.yml project file:

       plugins:
         loaders:
         - name: target-bigquery
           namespace: target_bigquery
           pip_url: target-bigquery
           executable: target-bigquery
           settings:
           - name: project_id
           - name: dataset_id
           - name: table_id
     

     To learn more about adding custom plugins, refer to the Plugin Management guide.

     Once you've got the loader working in your project, please consider contributing its description to the index of discoverable plugins so that it can be supported out of the box for new users!

   • If a Singer target for your data source doesn’t exist yet, learn how to build your own target by following Singer’s “Developing a Target” guide.

   Once you’ve got your new target project set up, you can add it to your Meltano project as a custom plugin by following the meltano add --custom instructions above. When asked to provide a pip install argument, you can provide a local directory path or Git repository URL.

4. Optionally, verify that the loader was installed successfully and that its executable can be invoked using meltano invoke:

    meltano invoke <plugin> --help
   
    # For example:
    meltano invoke target-postgres --help
   

   If you see the loader’s help message printed, the plugin was definitely installed successfully, but an error message related to missing configuration or an unimplemented --help flag would also confirm that Meltano can invoke the plugin’s executable.

Configure the loader #

Chances are that the loader you just added to your project will require some amount of configuration before it can start loading data.

To learn more about managing the configuration of your plugins, refer to the Configuration guide.

loaders:
- name: target-example
  config: {
    "setting": "value",
    "another_setting": true
  }

1. Find out what settings your loader supports using meltano config <plugin> list:

    meltano config <plugin> list
   
    # For example:
    meltano config target-postgres list
   

2. Assuming the previous command listed at least one setting, set appropriate values using meltano config <plugin> set:

    meltano config <plugin> set <setting> <value>
   
    # For example:
    meltano config target-postgres set host localhost
    meltano config target-postgres set port 5432
    meltano config target-postgres set user meltano
    meltano config target-postgres set password meltano
    meltano config target-postgres set dbname warehouse
    meltano config target-postgres set default_target_schema public
   

   You can turn on a local postgres docker instance with these configs using docker run --name postgres -e POSTGRES_PASSWORD=meltano -e POSTGRES_USER=meltano -e POSTGRES_DB=warehouse -d -p 5432:5432 postgres.

   This will add the non-sensitive configuration to your meltano.yml project file:

    plugins:
      loaders:
      - name: target-postgres
        variant: transferwise
        pip_url: pipelinewise-target-postgres
        config:
          host: localhost
          port: 5432
          user: meltano
          dbname: warehouse
          default_target_schema: public
   

   Sensitive configuration (like password) will instead be stored in your project’s .env file so that it will not be checked into version control:

    export TARGET_POSTGRES_PASSWORD=meltano
   

3. Optionally, verify that the configuration looks like what the Singer target expects according to its documentation using meltano config <plugin>:

    meltano config <plugin>
   
    # For example:
    meltano config target-postgres
   

   This will show the current configuration:

    {
      "host": "localhost",
      "port": 5432,
      "user": "meltano",
      "password": "meltano",
      "dbname": "warehouse",
      "ssl": "false",
      "default_target_schema": "public",
      "batch_size_rows": 100000,
      "flush_all_streams": false,
      "parallelism": 0,
      "parallelism_max": 16,
      "add_metadata_columns": false,
      "hard_delete": false,
      "data_flattening_max_level": 0,
      "primary_key_required": true,
      "validate_records": false
    }
   

Run a data integration (EL) pipeline #

Now that your Meltano project, extractor, and loader are all set up, we’ve reached the final chapter of this adventure, and it’s time to run your first data integration (EL) pipeline!

To learn more about data integration, refer to the Data Integration (EL) guide.

There’s just one step here: run your newly added extractor and loader in a pipeline using meltano elt:

meltano elt <extractor> <loader> --job_id=<pipeline name>

# For example:
meltano elt tap-gitlab target-postgres --job_id=gitlab-to-postgres

If everything was configured correctly, you should now see your data flow from your source into your destination! Check your postgres instance for the tables warehouse.schema.commits and warehouse.schema.tags.

If the command failed, but it’s not obvious how to resolve the issue, consider enabling debug mode to get some more insight into what’s going on behind the scenes. If that doesn’t get you closer to a solution, learn how to get help with your issue.

If you run meltano elt another time with the same Job ID, you’ll see it automatically pick up where the previous run left off, assuming the extractor supports incremental replication.

# Example
meltano elt tap-gitlab target-postgres --job_id=gitlab-to-postgres --dump=state > state.json

There is also a beta meltano run command which allows you to execute the same EL pipelines in a much more flexible fashion. This command allows you to chain multiple EL pipelines and add in other plugins inline too:

meltano run <extractor> <loader> <other_plugins>

# For example:
meltano run tap-gitlab target-postgres
meltano run tap-gitlab target-postgres dbt:test dbt:run

Or directly using the meltano invoke, which requires more settings to be defined prior to running

Next steps #

Now that you’ve successfully run your first data integration (EL) pipeline using Meltano, you have a few possible next steps:

• Schedule pipelines to run regularly
• Transform loaded data for analysis
• Containerize your project
• Deploy your pipelines in production

Schedule pipelines to run regularly #

Most pipelines aren’t run just once, but over and over again, to make sure additions and changes in the source eventually make their way to the destination.

To help you realize this, Meltano supports scheduled pipelines that can be orchestrated using Apache Airflow.

To learn more about orchestration, refer to the Orchestration guide.

1. Schedule a new meltano elt pipeline to be invoked on an interval using meltano schedule:

meltano schedule <pipeline name> <extractor> <loader> <interval>

# For example:
meltano schedule gitlab-to-postgres tap-gitlab target-postgres @daily

The pipeline name argument corresponds to the --job_id option on meltano elt, which identifies related EL(T) runs when storing and looking up incremental replication state.

To have scheduled runs pick up where your earlier manual run left off, ensure you use the same pipeline name.

This will add the new schedule to your meltano.yml project file:

schedules:
- name: gitlab-to-postgres
  extractor: tap-gitlab
  loader: target-postgres
  transform: skip
  interval: '@daily'

1. Optionally, verify that the schedule was created successfully using meltano schedule list:

    meltano schedule list
   

2. Add the Apache Airflow orchestrator to your project using meltano add, which will be responsible for managing the schedule and executing the appropriate meltano elt commands:

      meltano add orchestrator airflow
   

   This will add the new plugin to your meltano.yml project file:

    plugins:
      orchestrators:
      - name: airflow
        pip_url: apache-airflow==1.10.14
   

   It will also automatically add a meltano elt DAG generator to your project’s orchestrate/dags directory, where Airflow will be configured to look for DAGs by default.

3. Start the Airflow scheduler using meltano invoke:

    meltano invoke airflow scheduler
   
    # Add `-D` to run the scheduler in the background:
    meltano invoke airflow scheduler -D
   

   As long as the scheduler is running, your scheduled pipelines will run at the appropriate times.

4. Optionally, verify that a DAG was automatically created for each scheduled pipeline by starting the Airflow web interface:

    meltano invoke airflow webserver
   
    # Add `-D` to run the scheduler in the background:
    meltano invoke airflow webserver -D
   

5. Create melty the Admin user for logging in.

    meltano invoke airflow users create --username melty \
    --firstname melty \
    --lastname meltano \
    --role Admin \
    --password melty \
    --email melty@meltano.com
   

   The web interface and DAG overview will be available at http://localhost:8080.

Transform loaded data for analysis #

Once your raw data has arrived in your data warehouse, its schema will likely need to be transformed to be more appropriate for analysis.

To help you realize this, Meltano supports transformation using dbt.

To learn about data transformation, refer to the Data Transformation (T) guide.

1. To install the dbt transformer to your project run:

    meltano add transformer dbt
   

2. Once dbt has been installed in your Meltano project you will see the /transform directory populated with dbt artifacts.

   These artifacts are installed via the dbt file bundle. For more about file bundles, refer to the Plugin File bundles.

   Now all you need to do is start writing your dbt models in the /transform/models directory. This usually consists of a source.yml file defining the source tables you will be referencing inside your dbt models.

   For example the /transform/models/tap_gitlab/source.yml below configures dbt sources from the postgres tables where our tap-gitlab ELT job output to.

   Create and navigate to the /transform/models/tap_gitlab directory to hold your dbt models:

    mkdir ./transform/models/tap_gitlab
    touch  ./transform/models/tap_gitlab/source.yml
   

   Add the following content to your new source.yml file:

    config-version: 2
    version: 2
    sources:
      - name: tap_gitlab
        schema: public
        tables:
          - name: commits
          - name: tags
   

   The organization of your dbt project is up to you but if you’d like to run a specific set of models as part of a Meltano ELT pipeline it can be done via meltano elt tap target --transform=run which requires the model directory to match the extractor’s name using snake_case (i.e. tap_gitlab) so it can automatically find your models. Running as part of a pipeline allows Meltano to simplify dbt configuration by inferring some of your settings based on the pipeline tap and target.

   See more in the Data Transformation (T) guide - transform in your ELT pipeline.

3. Then add a model file with your SQL transformation logic. For example the dbt model SQL below generates a table with new commits in the last 7 days /transform/models/tap_gitlab/commits_last_7d.sql.

   Create your model file:

    touch  ./transform/models/tap_gitlab/commits_last_7d.sql
   

   Add the following content to your new commits_last_7d.sql file:

       
    {{
      config(
        materialized='table'
      )
    }}
   
    select *
    from {{ source('tap_gitlab', 'commits') }}
    where created_at::date >= current_date - interval '7 days'
       
   

4. Run your dbt models either using a pipeline transform:

    meltano elt <extractor> <loader>  --transform=run --job_id=<pipeline name>
   
    # For example:
    meltano elt tap-gitlab target-postgres --transform=run --job_id=gitlab-to-postgres
   

   Or alternatively you can run dbt directly using the meltano invoke, which requires more settings to be defined prior to running:

   • First add the following configs to your dbt settings:

       meltano config dbt set target postgres
       meltano config dbt set source_schema public
     

   • Then add the following env config, which sets environment variables at runtime, to your dev environment in the meltano.yml file.

       environments:
       - name: dev
         config:
           ...
         env:
           PG_ADDRESS: localhost
           PG_PORT: '5432'
           PG_USERNAME: meltano
           PG_DATABASE: warehouse
     

   • And finally add the postgres password to your .env file so that it doesnt get checked into git:

       PG_PASSWORD="meltano"
     

   • After these configurations are set you can run the dbt models using invoke:

       meltano invoke dbt:<command>
     
       # For example:
       meltano invoke dbt:run
     

   There is also a beta meltano run command which allows you to execute dbt in the same way as invoke but in a much more flexible fashion. This allows for inline dbt execution and more advanced reverse ETL use cases:

    meltano run <extractor> <loader> <other_plugins>
   
    # For example:
    meltano run tap-gitlab target-postgres dbt:test dbt:run tap-postgres target-gsheet
   

   After your transform run is complete you should see a new table named after your model warehouse.analytics.commits_last_7d in your target.

   See the transformer docs from other supported dbt commands like dbt:test, dbt:seed, dbt:snapshot and selection criteria like dbt:run --models tap_gitlab.*.

Containerize your project #

To learn how to containerize your project, refer to the Containerization guide.

Deploy your pipelines in production #

To learn how to deploy your pipelines in production, refer to the Deployment in Production guide.