Meltano v2.0 is here! Read about the release on our blog.
Discoverable plugins that are supported out of the box are available in Meltano Hub.
If you’ve added a custom plugin (or variant) to your project that could be discoverable and supported out of the box for new users, please contribute its description to Meltano Hub to save the next user the hassle of setting up the custom plugin. GitHub makes it easy to contribute changes without requiring you to leave your browser.
Discoverable plugin definitions in Meltano Hub have the a very similar format as custom plugin definition in your meltano.yml project file, so a copy-paste is usually sufficient.
The format and further requirements are laid out in more detail in the Meltano Hub plugin definition syntax document.
When the maintainer of the default variant of a discoverable plugin becomes unresponsive to issues and contributions filed by the community, that plugin is considered up for adoption, which means that we are looking for a different variant of the plugin with a more engaged maintainer to become the new default.
This new variant can either be a fork of the original default variant, or an alternative implementation for the same source or destination, as long as it is actively maintained.
If you maintain or are aware of such a variant, please add it to your Meltano project as a custom plugin and make it discoverable, or file an issue so that the Meltano core team can assist you.
As a plugin’s primary maintainer, you do not have to spend a lot of time improving the plugin yourself. In fact, attracting more users and getting the community involved is likely to recude your personal maintenance burden, since you’ll receive contributions with bug fixes and new features that you will only be expected to review, not build yourself.
discovery.yml #
When you need to make changes to discovery.yml, these changes are not automatically detected inside of the meltano repo during development. While there are a few ways to solve this problem, it is recommended to create a symbolic link in order ensure that changes made inside of the meltano repo appear inside the Meltano project you initialized and are testing on.
discovery.yml in the repo/Users/bencodezen/Projects/meltano/src/meltano/core/bundle/discovery.ymlOpen your Meltano project in your terminal
Create a symbolic link by running the following command:
ln -s $YOUR_DISCOVERY_YML_PATH
Now, when you run the ls -l command, you should see something like:
bencodezen staff 72 Nov 19 09:19 discovery.yml -> /Users/bencodezen/Projects/meltano/src/meltano/core/bundle/discovery.yml
Now, you can see your changes in discovery.yml live in your project during development! 🎉
discovery.yml version #
Whenever new functionality is introduced that changes the schema of discovery.yml (the exact properties it supports and their types), the version in src/meltano/core/bundle/discovery.yml and the VERSION constant in src/meltano/core/plugin_discovery_service.py must be incremented, so that older instances of Meltano don’t attempt to download and parse a discovery.yml its parser is not compatible with.
Changes to discovery.yml that only use existing properties do not constitute schema changes and do not require version to be incremented.
Watch “How taps are built” for an explanation of how Singer taps (which form the basis for Meltano extractors) work, and what goes into building new ones or verifying and modifying existing ones for various types of APIs.
Then watch “How transforms are built” for an explanation of how DBT transforms work, and what goes into building new ones for new data sources.
We should be good citizen about these, and use the default workflow to contribute. Most of these are on GitHub so:
meltano/meltano pipeline.We qualify taps with the capabilities it supports:
--properties format for the catalog--catalog format for the catalogYou should look at the tap’s documentation to see which one is supported.
Try to run the tap with the --discover switch, which should output a catalog on STDOUT.
STATE messages.target-postgres, then validate that:
This might be a configuration issue with the catalog file that is sent to the tap. Take a look at the tap’s documentation and look for custom metadata on the catalog.
When you need to expose data to the user through Meltano UI, this often will require updating the transforms and models. At a high level:
You can test local transforms in a project by adding them in a Meltano project’s transform > models > my_meltano_project directory.
Every transform file is a SQL file that will determine how the table is created. Some caveats include:
analytics.gitlab_issues), the syntax uses ref to refer to tables* seems to crash dbt. Instead, you should explicitly define every column. For example:users.user_id as user_id,
users.user_name as user_name,
issues.month_closed as month_closed,
issues.year_closed as year_closed,
Once you’ve created your transforms, you can run it with the following command:
# Replace your extractors / targets with the appropriate ones
meltano elt tap-gitlab target-postgres --transform only
To create a file bundle plugin like https://github.com/meltano/files-dbt, follow these steps:
files-<service/tool> (e.g. files-airflow or files-docker).setup.py, README.md, and LICENSE from https://github.com/meltano/files-dbt and edit these files as appropriate.bundle directory with an empty __init__.py file.bundle directory. All of these files will be copied over into the Meltano project directory when the file bundle is added to the project.bundle to the package_data["bundle"] array in setup.pyhttps://github.com/meltano/files-....src/meltano/core/bundle/discovery.yml under files. Set name and pip_url as appropriate, and if applicable, set namespace to the namespace of the plugin the file bundle is related to (e.g. dbt).meltano upgrade is run, add an update object with [file path]: True entries for each file.discovery.yml (and an appropriate CHANGELOG item, of course).