Skip to main content
Peliqan has ETL built-in for a wide range of connectors. This means that when you add a connection to a source, Peliqan will automatically create pipelines and start syncing the data into the Peliqan built-in data warehouse (DWH) or your own DWH. However, for some cases it might be useful to implement a custom pipeline in Peliqan, using low-code Python scripts.

​
Fetch data from a source

A custom pipeline will typically fetch data from some source incrementally: Incremental processing from source

​
Write data to the DWH

Next, the data from the source will be written to the DWH. Peliqan has a powerful built-in function dw.write() that will write a dataset to a table in the DWH, and prior to that it will create the table if needed and add columns if needed. More info: Writing data to tables

​
Handling deleted rows

A regular ETL pipeline cannot detect if records were deleted in the source. This means that when a record is deleted in a source, it will remain in the DWH. Here are various patterns to handle deleted records accordingly: Handling deleted rows

​
Creating history tables

You can set up a custom ETL pipeline to store historic data. More info: Slow changing dimensions (history tables)