Azure Databricks: Setup and Use in Customer Projects (A Practical Guide for FDEs)

What Is Azure Databricks?

Azure Databricks is, in Microsoft's words, a unified, open analytics platform for building, deploying, sharing and maintaining enterprise-grade data, analytics and AI solutions at scale. Microsoft says it integrates with cloud storage and security in your cloud account, and manages and deploys cloud infrastructure for you.

Its central idea is the data lakehouse. Microsoft describes a lakehouse as combining enterprise data warehouses and data lakes to accelerate, simplify and unify enterprise data solutions, with engineers, scientists, analysts and production systems working from a single source of truth.

What You Can Do With It

According to Microsoft's overview, Azure Databricks brings together:

• ETL and data engineering using Apache Spark with Delta and custom tools, with SQL, Python and Scala for pipeline logic. Auto Loader incrementally loads data from cloud object storage, and Lakeflow pipelines manage dependencies between datasets.
• Notebooks supporting Python, R and Scala as well as SQL, with visualisations next to markdown.
• SQL warehouses, scalable compute that administrators configure so end users can query without managing cloud complexity.
• Machine learning and AI, including MLflow, a machine learning runtime, and AI Functions that let SQL analysts call large language models from pipelines.
• Governance through Unity Catalog.
• Jobs and workflows that schedule notebooks, SQL queries and other code, with Git folders to sync projects with git providers.

For FDEs, that breadth is the appeal and the risk. One platform can serve many teams, but it also needs deliberate boundaries.

Setting Up Azure Databricks for a Customer: Seven Decisions

1. Agree the scope and the owners

Before any setup, settle which use case you are serving, who owns the platform, which data it will hold and which teams will use it. Ask who approves access and who pays. These questions belong in discovery; see the FDE project lifecycle.

2. Decide the network design first

By default, Databricks manages the network for classic compute resources. Many enterprises need more control, and Microsoft documents VNet injection, which deploys Azure Databricks classic compute resources into your own Azure virtual network. Microsoft says this enables network customisation, secure connectivity to Azure services and on-premises data sources, and traffic inspection.

What it allows, per the documentation: private connectivity to Azure services through service endpoints or private endpoints, on-premises access through user-defined routes, traffic inspection with network virtual appliances, custom DNS and egress control with additional network security group rules.

Details to plan around:

• Each workspace needs two dedicated subnets, and Databricks does not recommend subnets smaller than /26.
• Subnets cannot be shared across workspaces or used for other Azure resources.
• Subnet CIDR ranges cannot be changed after deployment. If subnets run out of space, Microsoft says to contact your Azure Databricks account team to request an increase.
• The person creating the workspace needs appropriate network permissions on the virtual network.

The takeaway: size and design the network before creating the workspace, and involve the customer's network team early. It is one of the few decisions that is painful to reverse.

3. Plan identity and access

Map the customer's groups to roles: data engineers, analysts, ML engineers, read-only business users. Use groups, not individuals, so access follows team changes. Microsoft notes that cloud administrators configure coarse access control permissions first, and Databricks administrators then manage finer ones, so agree who does what.

4. Set up governance with Unity Catalog

Microsoft describes Unity Catalog as the unified governance layer for data and AI built into Azure Databricks. Once enabled, it enforces access control, tracks lineage and logs activity. It is automatically enabled for workspaces created after 9 November 2023, while older workspaces must be upgraded.

Key concepts:

• Three-level namespace. Tables, views, volumes, functions and models are addressed as catalog.schema.object.
• Managed vs external. Managed tables and volumes have governance and storage lifecycle handled by Unity Catalog; external ones are governed only.
• Access controls. Privileges, attribute-based policies, and row and column filters.
• Lineage and audit. Lineage flows from source data through to models, services and dashboards, and an audit log system table records data access and system activity.

An illustrative grant, adapted for explanation and not run against a live workspace:

sql

CREATE CATALOG IF NOT EXISTS claims;

CREATE SCHEMA IF NOT EXISTS claims.curated;

GRANT SELECT ON SCHEMA claims.curated TO `analysts`;

A common design is one catalog per environment or domain, with schemas for raw, cleaned and curated data. That layout is practitioner convention, not a Microsoft requirement. Our guide to AI governance for forward deployed engineers covers the wider accountability picture.

5. Design the storage layout

Delta Lake is the optimised storage layer that provides the foundation for tables in a lakehouse on Databricks, and Microsoft says it is the default format: unless specified otherwise, all tables on Azure Databricks are Delta Lake tables. It extends Parquet files with a transaction log, which gives you:

• ACID transactions, so readers do not see half-written data.
• Time travel, because each write creates a new table version you can query or review.
• Schema enforcement, validating data on write against the table's schema.
• Schema evolution, updating a table schema without rewriting data.
• Batch and streaming on a single copy of data.

Microsoft warns against directly editing data or transaction log files in Delta directories, since that can corrupt tables. Treat tables as the interface.

Many teams organise layers as raw, cleaned and curated (often called bronze, silver and gold). It is convention, not a rule. The cleaning step is covered in data cleaning for AI projects, and the curated layer often follows the dimensional ideas in data modeling and star schema.

6. Match compute to the workload

Different users need different compute. Analysts running dashboards want SQL warehouses. Pipelines want scheduled job compute that starts, runs and stops. Interactive notebooks suit exploration. Keep these separate so a heavy experiment does not slow a production pipeline, and agree cost controls such as automatic shutdown, sensible sizing and tagging by team, using the options in the current documentation.

7. Automate deployment, scheduling and monitoring

Use jobs to schedule notebooks, SQL and other code. Microsoft also describes declarative bundles to define, deploy and run jobs and pipelines programmatically, and Git folders to sync with source control. Treat notebooks and pipeline definitions as code: version them, review changes and promote through test to production. Set alerts on failures and on runs that finish with no data.

Where Azure Databricks Fits Next to Azure Data Factory

The two are often used together. A common split, which is practitioner framing rather than a Microsoft rule, is to use Azure Data Factory to pull data from many sources into storage and to orchestrate, while Databricks handles heavier transformation, analytics and machine learning on that data. Databricks can also ingest directly, for example with Auto Loader from cloud storage. Pick one orchestrator per pipeline to avoid two schedulers fighting over the same job.

If the customer already runs Palantir Foundry, the question becomes overlap and ownership, not either/or. Our guide to AI data pipelines for messy enterprise data covers the platform-neutral pipeline stages.

How FDEs Use Azure Databricks for AI Work

• Feature and training data. Build clean, governed tables that models and assistants read from, with lineage back to source.
• Model development. MLflow and the machine learning runtime support experiments and model tracking; Unity Catalog governs models as securable objects.
• LLM in the pipeline. AI Functions let SQL analysts call language models from pipelines, which suits classification or extraction on text columns. Test accuracy and cost on a sample first.
• Serving analytics to the business. SQL warehouses expose curated tables to dashboards and tools.

Role boundaries vary by customer. See FDE vs MLOps engineer for how the work divides.

A Worked Scenario: A Claims Assistant for an Insurer

This is a hypothetical example to show the approach; the company and details are invented.

A mid-sized Indian insurer wants an assistant that helps claims handlers find similar past claims and flags missing documents. Data sits in a claims system, a document store and a policy database.

Network. The customer's security team requires private connectivity, so the FDE and the customer's network engineers design the virtual network and subnets before creating the workspace, leaving room for growth.

Ingestion. Azure Data Factory lands nightly extracts in storage. Databricks Auto Loader picks up new files incrementally.

Governance. A claims catalog has raw, cleaned and curated schemas. Handlers' groups get read access to curated tables; only a small group sees personal fields, with column filters for the rest.

Transformation. Jobs clean and standardise claims, resolve duplicate customers and publish a curated claims table. A feature table supports similarity search.

Assistant. The application reads curated data, and the FDE records which tables and versions feed it so any answer can be traced.

Operations. Jobs run on schedule with alerts. Lineage and audit logs help answer the customer's compliance questions, and a monthly review checks cost by team.

Common Mistakes

• Creating the workspace before the network design. Subnet ranges cannot be changed later.
• Skipping Unity Catalog planning. Retrofitting governance is harder than starting with it.
• One cluster for everything. Separate interactive, job and SQL workloads.
• Editing Delta files directly. Use tables and supported commands.
• Notebooks as production. Version, review and deploy as code.
• No cost visibility. Tag and review spend by team early.
• Treating the platform as the project. The goal is a working customer outcome, not a perfect lakehouse.

Skills to Build

Spark and SQL fundamentals, Python, Delta Lake concepts, access control and governance, cloud networking basics and clear communication with security and data teams. If you are moving from a data background, see data engineer to forward deployed engineer, and for the larger system around the platform, read how FDEs build enterprise AI platforms.

TL;DR: Azure Databricks is Microsoft's managed version of the Databricks lakehouse platform: one place to run data engineering, SQL analytics and machine learning on shared data. For a forward deployed engineer, success depends less on clicking "create workspace" and more on early decisions: network design (hard to change later), Unity Catalog governance, storage layout, who gets which compute, and how jobs are deployed and monitored. Get those right and the platform becomes a stable base for the customer's AI features