Skip to main content
Data sync are patterns to sync data between a source and a target. In many cases the source and target are both SaaS applications, for example keeping Customer records in sync between a CRM and an accounting system. Data syncs are implemented in Peliqan by writing low-code Python scripts (“Data apps”) that are scheduled to run for example every 5 minutes (near real-time), every hour or every day. These scripts are also referred to as “Custom data pipelines” and they use the Peliqan SaaS API wrapper functions to do so called writebacks to a target.
You can also use the Peliqan Reverse ETL app, which allows setting up Reverse ETL flows with field mapping, using a no-code UI. For more information visit:Reverse ETL

​
Patterns for data sync

Peliqan provides a wide range of features to implement robust, stable and efficient data syncs, by offering a set of patterns to implement these data syncs. The basic pattern is to read data from a Source, and write the data in destination or Target. From the Source, the data needs to be read incrementally. In the Target, the data needs to be written using an upsert. Data sync patterns

​
Data warehouse as central hub for data syncs

The patterns in Peliqan use the data warehouse (DWH) as a central data hub, which means the source is usually the data warehouse (which holds the data from the source SaaS application). Below is an example that syncs data from a CRM source to an Accounting target. First you add a connection to the source in Peliqan. This will automatically create an ETL pipeline and sync the data from the source into the DWH. Your data sync scripts read from the DWH tables and write to the target. Data sync with DWH as central data hub Note that Peliqan will also sync the data from the target to the DWH (in grey in the picture). This allows for verification of the data in the target, e.g. check if all additions or updates were executed correctly, if no data is missing etc.
It is important to notice that not every pattern can be used for every source or destination because of the limitations of the APIs from the connected platforms. However, because Peliqan uses the data warehouse as the central hub, the patterns are more generic and much more robust compared to direct data syncs between A and B, which are typical for no-code iPaaS automation platforms (without a DWH).

​
More info

Incremental processing from source

Read data incrementally from the source

Avoiding duplicates (upsert in target)

Prevent duplicates when writing to targets

Keeping track with link tables

Use link tables to track sync state

Parent child relations

Handle parent-child relationships in syncs