Azure Data Factory: Building Customer Data Pipelines (A Practical Guide for FDEs)

Microsoft notes that if either the source or sink linked service points to a self-hosted IR, the copy activity runs on that self-hosted IR. One logical self-hosted IR can span multiple machines in active-active mode for high availability.

For FDEs, the self-hosted IR is the usual answer to "the data is behind the customer's firewall". Because it makes outbound connections only, you generally avoid asking the customer to open inbound ports, which eases security reviews.

A Reliable Pattern for Customer Data Pipelines

ADF gives you the parts; the design is yours. This pattern is practitioner framing, not a Microsoft standard.

1. Land the raw data first

Copy source data into storage untouched, with a load date and source name. Raw copies let you replay, audit and debug without hitting the source system again.

2. Standardise and clean in a separate step

Fix types, formats and obvious errors after landing, not during the copy. See our guide to data cleaning for AI projects.

3. Resolve and publish shared entities

Match customers and products across systems and build trusted tables. See master data management and data modeling and star schema for the design behind the published layer.

4. Validate before you publish

Check row counts, null rates, key uniqueness and freshness. Fail the run or quarantine bad rows instead of silently publishing broken data.

5. Parameterise everything

Use parameters for source names, paths, dates and environments, so one pipeline serves many tables and moves from test to production unchanged. In a metadata-driven design, a control table lists what to load and a For-each loop processes it.

6. Load incrementally and make reruns safe

Full reloads are simple but slow and expensive at scale. A common approach is a watermark: store the latest modified date you loaded, then fetch only newer rows next time.

sql

SELECT *

FROM orders

WHERE last_modified > :last_watermark

  AND last_modified <= :run_start;

Pair this with idempotent writes: running the same load twice gives the same result, because you merge or overwrite by key instead of appending blindly. A failed run can then be repeated without duplicates.

Security Essentials

Microsoft's documentation on data movement security covers these points:

• Encryption in transit. Transfers between Data Factory and cloud data stores that support HTTPS or TLS use a secure channel, and connections to Azure SQL Database and Azure Synapse Analytics require encryption. For self-hosted IR transfers, Microsoft suggests IPsec VPN or ExpressRoute for extra protection of the on-premises to Azure channel.
• Encryption at rest. Data Factory stores no temporary data, cache or logs except encrypted linked service credentials for cloud data stores. Microsoft recommends enabling each data store's own encryption at rest.
• Secrets. Store data store credentials in Azure Key Vault, so Data Factory retrieves them when an activity runs and secrets are not pasted into pipeline definitions.
• Firewalls. A self-hosted IR needs outbound TCP 443, and some stores require allow-listing its IP address.
• Access control. Microsoft lists Microsoft Entra ID integration and role-based access control among the service's security features.

Beyond the platform, use least-privilege accounts for each source, separate environments and a record of what data moves where. Our guide to AI governance for forward deployed engineers covers accountability. If the data is regulated, involve the customer's compliance team early.

Monitoring, CI/CD and Operations

Unwatched pipelines fail quietly. Microsoft says ADF supports monitoring through Azure Monitor, an API, PowerShell, Azure Monitor logs and health panels in the Azure portal, and supports CI/CD for pipelines using Azure DevOps and GitHub.

Practical operations checklist:

• Alert on failures and on runs that finish but load zero rows.
• Track freshness: when was each published table last updated?
• Keep definitions in source control and promote through test to production.
• Document each pipeline's owner, schedule and consumers.
• Plan for source changes: a renamed upstream column should fail validation loudly, not corrupt data downstream.

Azure Data Factory vs Fabric Data Factory

Microsoft calls Fabric Data Factory the next generation of ADF. According to its comparison page, key differences include:

The page also says approximately 90% of ADF activities are already available in Fabric, and that new Fabric features are not backported to ADF. Our inference, not a Microsoft recommendation: choose Fabric for new, Fabric-centred work, and stay on ADF where a customer depends on SSIS, managed virtual networks or other capabilities not yet in Fabric. Check the current page, because the gaps change.

A Worked Scenario: Feeding a Customer Assistant

This is a hypothetical example to show the approach; the company and details are invented.

A mid-sized Indian distributor wants an assistant that answers account questions for its sales team. Orders sit in an on-premises ERP, contacts in a cloud CRM and tickets in a helpdesk SaaS. The customer already uses Azure.

Reach the data. The FDE installs a self-hosted integration runtime on a VM inside the customer's network for the ERP. The CRM and helpdesk use the Azure IR. Credentials live in Key Vault, with read-only service accounts per system.

Land and standardise. A parameterised pipeline copies each table nightly into a raw zone, then a second step standardises dates, currencies and phone formats.

Load incrementally. Orders use a last-modified watermark. The first run does a full load; later runs fetch changes and merge by order ID, so reruns are safe.

Publish. A matching step links the same customer across the three systems, and a published customer table feeds the assistant.

Operate. Alerts fire on failures and zero-row loads. A freshness check blocks publication if the ERP extract is over a day old, and the assistant says when data is stale.

An agent built on the platform described in Copilot Studio for FDEs could then read from the published tables.

Common Mistakes

• Transforming during the copy. Land raw data first so you can replay.
• Full reloads forever. Go incremental before data grows.
• Non-idempotent writes. Reruns create duplicates.
• Secrets in pipeline definitions. Use Key Vault.
• No validation or alerts. Silent failures corrupt downstream AI.
• Choosing the platform by habit. Compare ADF and Fabric for the actual customer.

Where This Fits in an FDE Career

Data integration is core to the FDE project lifecycle: discovery uncovers the systems, then pipelines make them usable. If you come from a data background, see data engineer to forward deployed engineer. For how the role differs from infrastructure-focused jobs, read forward deployed engineer vs cloud engineer. Related automation ideas appear in AI workflow automation for forward deployed engineers.

TL;DR: Azure Data Factory (ADF) is Microsoft's managed cloud service for building and scheduling data pipelines that move and transform data at scale. For a forward deployed engineer, it is often the tool already sitting in a customer's Azure environment. Build pipelines that land raw data first, load incrementally, run safely with parameters and stored secrets, reach on-premises systems through a self-hosted integration runtime, and are monitored from day one. For new projects, check whether Fabric's Data Factory is the better starting point, since Microsoft calls it the next generation.

‍

Frequently Asked Questions

Become one of India’s first Forward-Deployed Engineers.

The world is hiring - and this Academy prepares you for it.

Apply Now