How to copy data from On-Prem PostgreSQL to Azure SQL Database use Data Factory

  1. Set up a self-hosted integration runtime on a server that has access to the On-premises PostgreSQL database. This integration runtime acts as a bridge between your on-premises data source and Azure Data Factory.
  2. Create a linked service for the On-premises PostgreSQL data source in the Azure Data Factory portal, and specify the integration runtime you set up in step 1 as the integration runtime for the linked service.
  3. Create a linked service for the Azure SQL Database destination in the Azure Data Factory portal.
  4. Create a pipeline in Azure Data Factory that includes a copy activity.
  5. In the copy activity, select the On-premises PostgreSQL data source as the source, and the Azure SQL Database destination as the sink.
  6. Configure the source and sink datasets with the necessary connection details and credentials.
  7. Map the columns in the source dataset to the corresponding columns in the sink dataset.
  8. Optionally, you can apply transformations to the data using Azure Data Factory’s built-in data transformation activities, such as mapping, aggregation, and filtering.
  9. Finally, you can run the pipeline to move the data from On-premises PostgreSQL to Azure SQL Database.

##JSON##
{
    "name": "CopyPostgreSQL2SQL",
    "properties": {
        "activities": [
            {
                "name": "CopyPostgreSQL2SQL",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "PostgreSqlSource"
                    },
                    "sink": {
                        "type": "AzureSqlSink",
                        "writeBehavior": "insert",
                        "sqlWriterUseTableLock": true,
                        "disableMetricsCollection": false
                    },
                    "enableStaging": false,
                    "translator": {
                        "type": "TabularTranslator",
                        "mappings": [
                            {
                                "source": {
                                    "name": "employee_id",
                                    "type": "Int32"
                                },
                                "sink": {
                                    "name": "employee_id",
                                    "type": "Int32",
                                    "physicalType": "int"
                                }
                            },
                            {
                                "source": {
                                    "name": "first_name",
                                    "type": "String"
                                },
                                "sink": {
                                    "name": "first_name",
                                    "type": "String",
                                    "physicalType": "varchar"
                                }
                            },
                            {
                                "source": {
                                    "name": "last_name",
                                    "type": "String"
                                },
                                "sink": {
                                    "name": "last_name",
                                    "type": "String",
                                    "physicalType": "varchar"
                                }
                            },
                            {
                                "source": {
                                    "name": "birth_date",
                                    "type": "DateTime"
                                },
                                "sink": {
                                    "name": "birth_date",
                                    "type": "DateTime",
                                    "physicalType": "date"
                                }
                            },
                            {
                                "source": {
                                    "name": "hire_date",
                                    "type": "DateTime"
                                },
                                "sink": {
                                    "name": "hire_date",
                                    "type": "DateTime",
                                    "physicalType": "date"
                                }
                            }
                        ],
                        "typeConversion": true,
                        "typeConversionSettings": {
                            "allowDataTruncation": true,
                            "treatBooleanAsNumber": false
                        }
                    }
                },
                "inputs": [
                    {
                        "referenceName": "PostgreSqlTable",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "Connect2AzSQLDB",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": [],
        "lastPublishTime": "2023-02-27T12:56:48Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

Leave a comment