Meltano v2.0 is here! Read about the release on our blog.

Settings

Meltano supports a number of settings that allow you to fine tune its behavior, which are documented here. To quickly find the setting you’re looking for, use the Table of Contents in the sidebar.

As described in the Configuration guide, Meltano will determine the values of these settings by first looking in the environment, then in your project’s .env file, and finally in your meltano.yml project file, falling back to a default value if nothing was found.

You can use meltano config meltano list to list all available settings with their names, environment variables, and current values.

Configuration that is not environment-specific or sensitive should be stored in your meltano.yml project file and checked into version control. Sensitive values like passwords and tokens are most appropriately stored in the environment or your project’s .env file.

meltano config meltano set <setting> <value>, which is used in the examples below, will automatically store configuration in meltano.yml or .env as appropriate.

If supported by the plugin type, its configuration can be tested using meltano config <plugin> test.

Plugin settings #

For plugin settings, refer to the specific plugin’s documentation (extractors, loaders), or use meltano config <plugin> list to list all available settings with their names, environment variables, and current values.

Your Meltano project #

These are settings specific to your Meltano project.

send_anonymous_usage_stats #

Meltano is open source software thats free for anyone to use. The best thing a user could do to give back to the community, aside from contributing code or reporting issues, is contribute anonymous usage stats to allow the maintainers to understand how features are being utilized ultimately helping the community build a better product.

By default, Meltano shares anonymous usage data with the Meltano team using Google Analytics and Snowplow. We use this data to learn about the size of our user base and the specific Meltano features they are using, which helps us determine the highest impact changes we can make in each release to make Meltano even more useful for you and others like you.

We also provide some of this data back to the community via MeltanoHub to help users understand the overall usage of plugins within Meltano.

If enabled, Meltano will use the value of the project_id setting to uniquely identify your project in Google Analytics. This project ID is also sent along when Meltano requests available plugins from the URLs identified by the hub_url or discovery_url setting.

If you’d like to send the tracking data to a different Google Analytics account than the one run by the Meltano team, the Tracking IDs can be configured using the tracking_ids.* settings below.

If you’d like to send the tracking data to a different Snowplow account than the one run by the Meltano team, the collector endpoints can be configured using the snowplow.collector_endpoints setting.

Meltano also tracks anonymous web metrics when browsing the Meltano UI pages.

See more about our anonymization standards and anonymous usage stats Q&A below for more details. Also refer to the Meltano data team handbook page for our “Philosophy of Telemetry”.

With all that said, if you’d still prefer to use Meltano without sending the maintainers this kind of data, you’re able to disable tracking entirely using one of these methods:

  • When creating a new project, pass --no_usage_stats to meltano init
  • In an existing project, set the send_anonymous_usage_stats setting to false
  • To disable tracking in all projects in one go, set the MELTANO_SEND_ANONYMOUS_USAGE_STATS environment variable to false

How to use # 

meltano config meltano set send_anonymous_usage_stats false

export MELTANO_SEND_ANONYMOUS_USAGE_STATS=false

meltano init --no_usage_stats demo-project

Anonymization Standards #

Unless otherwise approved, any user-entered data is anonymized client-side before being submitted to Meltano. This section describes which data is sent in clear text and which data is obfuscated via one-way hashing.

We capture these in clear text:

  • plugin names
  • plugin variant names
  • command names
  • execution context, such as:
    • OS version
    • Python version
    • project ID

We anonymize these with one-way hashing before reporting:

  • CLI args
  • plugin config

These items will never be collected or reported back to meltano:

  • your settings values
  • your secrets or credentials
  • the contents of your meltano.yml file

Anonymous Usage Stats Q&A #

Q: What is a one way hash and how is it helpful? #

A:

One-way hashing is a way of obfuscating sensitive data such that:

  1. The same input value always produces the same output value (aka “hash”).
  2. The results are mathematically and statistically extremely difficult (read: near impossible) to reverse engineer back to the source value.
  3. Hash results are extremely helpful in safely and anonymously detecting changes to a file or configuration. Without passing the entire configuration, and without providing a hacker any means of decoding/decrypting the data back to its source, we can see that a file (such as meltano.yml) has not changed since its last hash was generated.
Q: Why does Meltano use hashing? #

A:

Meltano hashes any fields at all which could be used by a hacker to compromise a project or user. We will never know what freeform text arguments you passed in via the command line, we won’t have any data at all which could be used to compromise your environment, and whatever data we collect, we’ll never sell, share, or trade your data with any third parties.

Should I enable or disable anonymous reporting? #

A:

We hope you will choose to enable reporting, because this really does help us - and it helps the Meltano community in a very real way.

If you still have any concerns about keeping anonymous reporting enabled, we hope you’ll share those concerns with us. You can do so by emailing hello@meltano.com or by logging an issue in our Meltano Issue Tracker.

project_id #

Used by Meltano to uniquely identify your project in Google Analytics if the send_anonymous_usage_stats setting is enabled.

How to use # 

meltano config meltano set project_id <randomly-generated-token>

export MELTANO_PROJECT_ID=<randomly-generated-token>

database_uri #

  • Environment variable: MELTANO_DATABASE_URI
  • meltano * CLI option: --database-uri
  • Default: sqlite:///$MELTANO_PROJECT_ROOT/.meltano/meltano.db

Meltano stores various types of metadata in a project-specific system database, that takes the shape of a SQLite database stored inside the .meltano directory at .meltano/meltano.db by default.

You can choose to use a different system database backend or configuration using the --database-uri option of meltano subcommands, or the MELTANO_DATABASE_URI environment variable.

How to use # 

meltano config meltano set database_uri postgresql://<username>:<password>@<host>:<port>/<database>

export MELTANO_DATABASE_URI=postgresql://<username>:<password>@<host>:<port>/<database>

meltano elt --database-uri=postgresql://<username>:<password>@<host>:<port>/<database> ...

Targeting a PostgreSQL Schema #

When using PostgreSQL as your system database, you can choose the target schema within that database by adding ?options=-csearch_path%3D<schema> directly to the end of your database_uri and MELTANO_DATABASE_URI.

You are also able to add multiple schemas, which PostgreSQL will work through from left to right until it finds a valid schema to target, by using ?options=-csearch_path%3D<schema>,<schema_two>

If you dont target a schema then by default PostgreSQL will try to use the public schema.

postgresql://<username>:<password>@<host>:<port>/<database>?options=-csearch_path%3D<schema>

database_max_retries #

This sets the maximum number of reconnection attempts in case the initial connection to the database fails because it isn’t available when Meltano starts up.

Note: This affects the initial connection attempt only after which the connection is cached. Subsequent disconnections are handled by SQLALchemy

How to use # 

meltano config meltano set database_max_retries 3

export MELTANO_DATABASE_MAX_RETRIES=3

database_retry_timeout #

This controls the retry interval (in seconds) in case the initial connection to the database fails because it isn’t available when Meltano starts up.

Note: This affects the initial connection attempt only after which the connection is cached. Subsequent disconnections are handled by SQLALchemy

How to use # 

meltano config meltano set database_retry_timeout 5

export MELTANO_DATABASE_RETRY_TIMEOUT=5

project_readonly #

Enable this setting to indicate that your Meltano project is deployed as read-only, and to block all modifications to project files through the CLI and UI in this environment.

Specifically, this prevents adding plugins or pipeline schedules to your meltano.yml project file, as well as modifying plugin configuration stored in meltano.yml or .env.

Note that meltano config <plugin> set and the UI can still be used to store configuration in the system database, but that settings that are already set in the environment or meltano.yml take precedence and cannot be overridden.

This setting differs from the ui.readonly setting in two ways:

  1. it does not block write actions in the UI that do not modify project files, like storing settings in the system database, and
  2. it also affects the CLI.

How to use # 

meltano config meltano set project_readonly true

export MELTANO_PROJECT_READONLY=true

hub_url #

Where Meltano can find the Hub that lists all discoverable plugins.

This manifest is primarily used by meltano discover and meltano add. It is also used in cases where the full plugin definition is needed but no lock artifact or cached discovery.yml is found.

How to use # 

meltano config meltano set hub_url http://localhost:4000

export MELTANO_HUB_URL=http://localhost:4000

discovery_url #

Where Meltano can find the discovery.yml manifest that lists all discoverable plugins that are supported out of the box.

This manifest is used by meltano discover and meltano add, among others.

To disable downloading the remote discovery.yml manifest and only use the project-local or packaged version, set this setting to false or any other string not starting with http:// or https://.

How to use # 

meltano config meltano set discovery_url https://meltano.example.com/discovery.yml
meltano config meltano set discovery_url false

export MELTANO_DISCOVERY_URL=https://meltano.example.com/discovery.yml
export MELTANO_DISCOVERY_URL=false

discovery_url_auth #

The value of the Authorization header sent when making a request to discovery_url.

No Authorization header is applied under the following conditions:

  • discovery_url_auth is not set
  • discovery_url_auth is set to false, null or an empty string

How to use # 

meltano config meltano set discovery_url_auth "Bearer $ACCESS_TOKEN"
meltano config meltano set discovery_url_auth false

export MELTANO_DISCOVERY_URL_AUTH="Bearer $ACCESS_TOKEN"
export MELTANO_DISCOVERY_URL_AUTH=false

meltano CLI #

These settings can be used to modify the behavior of the meltano CLI.

cli.log_level #

  • Environment variable: MELTANO_CLI_LOG_LEVEL, alias: MELTANO_LOG_LEVEL
  • meltano CLI option: --log-level
  • Options: debug, info, warning, error, critical
  • Default: info

The granularity of CLI logging. Ignored if a local logging config is found.

How to use # 

meltano config meltano set cli log_level debug

export MELTANO_CLI_LOG_LEVEL=debug
export MELTANO_LOG_LEVEL=debug

meltano --log-level=debug ...

cli.log_config #

  • Environment variable: MELTANO_CLI_LOG_CONFIG, alias: MELTANO_LOG_CONFIG
  • meltano CLI option: --log-config
  • Default: logging.yaml

The path of a valid yaml formatted python logging dict config file to use to configure logging if present.

How to use # 

meltano config meltano set cli log_config /path/to/logging.yaml

export MELTANO_CLI_LOG_CONFIG=/path/to/logging.yaml
export MELTANO_LOG_CONFIG=/path/to/logging.yaml

meltano --log-config=/path/to/logging.yaml ...

A sample logging config:

version: 1
disable_existing_loggers: false

formatters:
  default:
    format: "[%(asctime)s] [%(process)d|%(threadName)10s|%(name)s] [%(levelname)s] %(message)s"
  structured_plain:
    (): meltano.core.logging.console_log_formatter
    colors: False
  structured_colored:
    (): meltano.core.logging.console_log_formatter
    colors: True
  key_value:
    (): meltano.core.logging.key_value_formatter
    sort_keys: False
  json:
    (): meltano.core.logging.json_formatter

handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: structured_colored
    stream: "ext://sys.stderr"
  file:
    class: logging.FileHandler
    level: INFO
    filename: /var/log/meltano.log
    formatter: json

root:
  level: DEBUG
  propagate: yes
  handlers: [console, file]

meltano elt #

These settings can be used to modify the behavior of meltano elt.

elt.buffer_size #

Size (in bytes) of the buffer between extractor and loader (Singer tap and target) that stores messages output by the extractor while they are waiting to be processed by the loader.

When an extractor generates messages (records) faster than the loader can process them, the buffer may fill up completely, at which point the extractor will be blocked until the loader has worked through enough messages to make half of the buffer size available again for new extractor output.

The length of a single line of extractor output is limited to half the buffer size. With a default buffer size of 10MiB, the maximum message size would therefore be 5MiB.

How to use # 

meltano config meltano set elt.buffer_size 52428800 # 50MiB in bytes

export MELTANO_ELT_BUFFER_SIZE=52428800

Meltano UI server #

These settings can be used to configure the Meltano UI server.

Meltano UI feature settings and customization settings have their own sections.

ui.bind_host #

The host to bind to.

Together with the ui.bind_port setting, this setting corresponds to Gunicorn’s bind setting.

How to use # 

meltano config meltano set ui bind_host 127.0.0.1

export MELTANO_UI_BIND_HOST=127.0.0.1

meltano ui --bind=127.0.0.1

ui.bind_port #

The port to bind to.

Together with the ui.bind_host setting, this setting corresponds to Gunicorn’s bind setting.

How to use # 

meltano config meltano set ui bind_port 80

export MELTANO_UI_BIND_PORT=80

meltano ui --bind-port=80

ui.server_name #

The host and port Meltano UI is available at, e.g. <host>:<port>.

The port will usually match the ui.bind_port setting, and can be omitted when the default port for HTTP (80) or HTTPS (443) is used.

Unless the ui.session_cookie_domain setting is set, this setting will be used as the session cookie domain.

If the ui.notification setting is enabled, this setting will be used to generate external URLs in notification emails.

When set, Meltano UI will only respond to requests whose hostname (Host header) matches this setting. If this is undesirable, you can set the ui.session_cookie_domain setting instead. This may be the case when Meltano UI is situated behind a load balancer performing health checks without specifying a hostname.

If the ui.authentication setting is enabled, meltano ui will print a security warning if neither this setting or the ui.session_cookie_domain setting has been set.

This setting corresponds to Flask’s SERVER_NAME setting.

How to use # 

meltano config meltano set ui server_name meltano.example.com

export MELTANO_UI_SERVER_NAME=meltano.example.com

meltano ui setup <server_name> can be used to generate secrets for the ui.secret_key and ui.password_salt settings, that will be stored in a your project’s .env file along with the specified server_name.

meltano ui setup meltano.example.com

The domain match rule that the session cookie will be valid for.

If not set, the cookie will be valid for all subdomains of the configured ui.server_name.

If the ui.authentication setting is enabled, meltano ui will print a security warning if neither this setting or the ui.server_name setting has been set.

This setting corresponds to Flask’s SESSION_COOKIE_DOMAIN setting.

How to use # 

meltano config meltano set ui session_cookie_domain meltano.example.com

export MELTANO_UI_SESSION_COOKIE_DOMAIN=meltano.example.com

Enable the Secure flag on the session cookie, so that the client will only send it to the server in HTTPS requests.

The application must be served over HTTPS for this to make sense.

This setting corresponds to Flask’s SESSION_COOKIE_SECURE setting.

How to use # 

meltano config meltano set ui session_cookie_secure true

export MELTANO_UI_SESSION_COOKIE_SECURE=true

ui.secret_key #

A secret key that will be used for securely signing the session cookie.

If the ui.authentication setting is enabled, meltano ui will print a security warning if this setting has not been changed from the default.

This setting corresponds to Flask’s SECRET_KEY setting.

How to use # 

meltano config meltano set ui secret_key <randomly-generated-secret>

export MELTANO_UI_SECRET_KEY=<randomly-generated-secret>

meltano ui setup <server_name> can be used to generate secrets for the this setting and ui.password_salt, that will be stored in your project’s .env file along with the specified ui.server_name.

meltano ui setup meltano.example.com

ui.password_salt #

The HMAC salt to use when hashing passwords.

If the ui.authentication setting is enabled, meltano ui will print a security warning if this setting has not been changed from the default.

This setting corresponds to Flask-Security’s SECURITY_PASSWORD_SALT setting.

How to use # 

meltano config meltano set ui password_salt <randomly-generated-secret>

export MELTANO_UI_PASSWORD_SALT=<randomly-generated-secret>

meltano ui setup <server_name> can be used to generate secrets for the this setting and ui.secret_key, that will be stored in your project’s .env file along with the specified ui.server_name.

meltano ui setup meltano.example.com

ui.workers #

The number of worker processes meltano ui will use to handle requests.

This setting corresponds to Gunicorn’s workers setting.

How to use # 

meltano config meltano set ui workers 1

export MELTANO_UI_WORKERS=1

ui.forwarded_allow_ips #

Comma-separated front-end (reverse) proxy IPs that are allowed to set secure headers to indicate HTTPS requests.

Set to * to disable checking of front-end IPs, which can be useful for setups where you don’t know in advance the IP address of front-end, but you still trust the environment.

This setting corresponds to Gunicorn’s forwarded_allow_ips setting.

How to use # 

meltano config meltano set ui forwarded_allow_ips "*"

export MELTANO_UI_FORWARDED_ALLOW_IPS="*"

Meltano UI features #

These settings can be used to enable certain features of Meltano UI.

Meltano UI server settings and customization settings have their own sectionss

ui.readonly #

To block all write actions in the Meltano UI, you can run it in in read-only mode.

If you’re enabling the ui.authentication setting and would like to only use read-only mode for anonymous users, enable the ui.anonymous_readonly setting instead.

This setting differs from the project_readonly setting in two ways:

  1. it also blocks write actions in the UI that do not modify project files, like storing settings in the system database, and
  2. it does not affect the CLI.

How to use # 

meltano config meltano set ui readonly true

export MELTANO_UI_READONLY=true

ui.authentication #

Use this setting to enable authentication and disallow anonymous usage of your Meltano instance.

Additionally, you will need to:

  1. Ensure your configuration is secure by setting the ui.secret_key and ui.password_salt settings, as well as ui.server_name or ui.session_cookie_domain, manually or using meltano ui setup <server_name>.

  2. Create at least one user using meltano user add.

How to use # 

meltano config meltano set ui authentication true

export MELTANO_UI_AUTHENTICATION=true

ui.anonymous_readonly #

When the ui.authentication setting is enabled, enabling this setting will allow anonymous users read-only access to Meltano UI. Once a user is authenticated, write actions will be available again.

This setting is especially useful when setting up a publicly available demo instance of Meltano UI for anonymous users to interact with. These users will not be able to make any changes, but admins will once they sign in.

How to use # 

meltano config meltano set ui anonymous_readonly true

export MELTANO_UI_ANONYMOUS_READONLY=true

ui.notification #

Meltano can send email notifications upon certain events.

Your outgoing mail server can be configured using the mail.* settings below.

To ease the development and testing, Meltano is preconfigured to use a local MailHog instance to trap all the outgoing emails.

Use the following docker command to start it:

docker run --rm -p 1025:1025 -p 8025:8025 --name mailhog mailhog/mailhog

All emails sent by Meltano should now be available at http://localhost:8025/

How to use # 

meltano config meltano set ui notification true

export MELTANO_UI_NOTIFICATION=true

Meltano UI customization #

These settings can be used to customize certain aspects of Meltano UI.

Meltano UI server settings and feature settings have their own sections.

ui.logo_url #

Customize the logo used by Meltano UI in the navigation bar and on the sign-in page (when the ui.authentication setting is enabled).

How to use # 

meltano config meltano set ui logo_url https://meltano.com/meltano-logo-with-text.svg

export MELTANO_UI_LOGO_URL=https://meltano.com/meltano-logo-with-text.svg

Mail server #

Meltano uses Flask-Mail to send emails. Take a look at the documentation to properly configure your outgoing email server.

mail.server # 

meltano config meltano set mail server smtp.example.com

export MELTANO_MAIL_SERVER=smtp.example.com

mail.port # 

meltano config meltano set mail port 25

export MELTANO_MAIL_PORT=25

mail.default_sender # 

meltano config meltano set mail default_sender '"Example Meltano" <bot@meltano.example.com>'

export MELTANO_MAIL_DEFAULT_SENDER='"Example Meltano" <bot@meltano.example.com>'

mail.use_tls # 

meltano config meltano set mail use_tls true

export MELTANO_MAIL_USE_TLS=true

mail.username # 

meltano config meltano set mail username meltano

export MELTANO_MAIL_USERNAME=meltano

mail.password # 

meltano config meltano set mail password meltano

export MELTANO_MAIL_PASSWORD=meltano

mail.debug # 

meltano config meltano set mail debug true

export MELTANO_MAIL_DEBUG=true

mail.sendgrid_unsubscribe_group_id #

If you are using the SendGrid SMTP API you may optionally set the SendGrid unsubscribe group ID.

meltano config meltano set mail sendgrid_unsubscribe_group_id 42

export MELTANO_MAIL_SENDGRID_UNSUBSCRIBE_GROUP_ID=42

OAuth Service #

Meltano ships with an OAuth Service to handle the OAuth flow in the Extractors’ configuration.

To run this service, you **must** have a registered OAuth application on the [Authorization server](https://www.oauth.com/oauth2-servers/definitions/#the-authorization-server).

Most importantly, the Redirect URI must be set properly so that the OAuth flow can be completed.

This process is specific to each Provider.

The OAuth Service is bundled within Meltano, and is automatically started with meltano ui and mounted at /-/oauth for development purposes.

As it is a Flask application, it can also be run as a standalone using:

FLASK_ENV=production FLASK_APP=meltano.oauth python -m flask run --port 9999

oauth_service.url #

The local OAuth service for development purposes is available at /-/oauth.

How to use # 

meltano config meltano set oauth_service url https://oauth.svc.meltanodata.com

export MELTANO_OAUTH_SERVICE_URL=https://oauth.svc.meltanodata.com

oauth_service.providers #

To enable specific providers, use comma-separated oauth.provider names from discovery.yml. To enable all providers, use all.

How to use # 

meltano config meltano set oauth_service providers facebook,google_adwords

export MELTANO_OAUTH_SERVICE_PROVIDERS=facebook,google_adwords

oauth_service.facebook.client_id # 

meltano config meltano set oauth_service facebook client_id <facebook-client-id>

export OAUTH_FACEBOOK_CLIENT_ID=<facebook-client-id>

oauth_service.facebook.client_secret # 

meltano config meltano set oauth_service facebook client_secret <facebook-client-secret>

export OAUTH_FACEBOOK_CLIENT_SECRET=<facebook-client-secret>

oauth_service.google_adwords.client_id # 

meltano config meltano set oauth_service google_adwords client_id <google-adwords-client-id>

export OAUTH_GOOGLE_ADWORDS_CLIENT_ID=<google-adwords-client-id>

oauth_service.google_adwords.client_secret # 

meltano config meltano set oauth_service google_adwords client_secret <google-adwords-client-secret>

export OAUTH_GOOGLE_ADWORDS_CLIENT_SECRET=<google-adwords-client-secret>

OAuth Single-Sign-On #

These variables are specific to Flask-OAuthlib and work with OAuth authentication with GitLab.

These settings are used for single-sign-on using an external OAuth provider.

For more information on how to get these from your GitLab application, check out the integration docs from GitLab.

oauth.gitlab.client_id # 

meltano config meltano set oauth gitlab client_id <gitlab-client-id>

export OAUTH_GITLAB_CLIENT_ID=<gitlab-client-id>
export OAUTH_GITLAB_APPLICATION_ID=<gitlab-client-id>

oauth.gitlab.client_secret # 

meltano config meltano set oauth gitlab client_secret <gitlab-client-secret>

export OAUTH_GITLAB_CLIENT_SECRET=<gitlab-client-secret>
export OAUTH_GITLAB_SECRET=<gitlab-client-secret>

Analytics Tracking IDs (deprecated, will be removed in a future version) #

Google Analytics Tracking IDs to be used if the send_anonymous_usage_stats setting is enabled.

tracking_ids.cli #

Tracking ID for usage of the meltano CLI.

meltano config meltano set tracking_ids cli UA-123456789-1

export MELTANO_TRACKING_IDS_CLI=UA-123456789-1

tracking_ids.ui #

Tracking ID for usage of Meltano UI.

meltano config meltano set tracking_ids ui UA-123456789-2

export MELTANO_TRACKING_IDS_UI=UA-123456789-2

tracking_ids.ui_embed #

Tracking ID for usage of Meltano UI’s Embed feature.

meltano config meltano set tracking_ids ui_embed UA-123456789-3

export MELTANO_TRACKING_IDS_UI_EMBED=UA-123456789-3

Snowplow Tracking #

snowplow.collector_endpoints #

Snowplow collector endpoints to be used if the send_anonymous_usage_stats setting is enabled. Events will be sent to all of these collectors.

Feature Flags #

ff.enable_uvicorn #

ff.env_var_strict_mode #

Causes an exception to be raised if an environment variable is used within the project’s Meltano configuration but that environment variable is not set.

Edit this page!