Discuss a project

Case studies / Selected project experience

Business problems. Practical solutions.

See how our founder has approached disconnected data, application integration and manual file transfers. These anonymised projects explain the business need first, followed by the technical approach.

01 / Lakehouse engineering

Bring Disconnected Data Together for Reporting

Anonymized case study
DatabricksdbtDelta LakeKafkaAWS / Azure

The business challenge

Different source systems need different ingestion behavior, but maintaining a bespoke pipeline for every feed makes onboarding, recovery and change difficult to manage.

The approach

Separate source configuration from reusable ingestion code. YAML files select the appropriate utility suite for APIs, SQL Server, cloud files or Kafka, and define the target bronze tables in Databricks. dbt then builds validated silver models and business-ready gold datasets.

Engineering scope

Source configuration templates, reusable ingestion utilities, Delta bronze tables, dbt silver and gold models, data tests, BI-facing datasets and operational runbooks.

How the data moves

Bring Disconnected Data Together for Reporting: source-to-consumption architectureMultiple sources: APIs · SQL Server · ADLS / S3 · Kafka. Configured ingestion: YAML selects source-specific utility code. Bronze: Raw Delta tables in Databricks. Silver → Gold: dbt validation, models and business logic. Consumption: BI reporting and analyticsAPIsSQL ServerADLS / S3KafkaIngestion utilitiesYAML configurationBronzeDatabricks / DeltaSilverdbt modelsGolddbt business logicBI reportingAnalyticsBring Disconnected Data Together for Reporting: source-to-consumption architectureMultiple sources: APIs · SQL Server · ADLS / S3 · Kafka. Configured ingestion: YAML selects source-specific utility code. Bronze: Raw Delta tables in Databricks. Silver → Gold: dbt validation, models and business logic. Consumption: BI reporting and analyticsAPIsSQL ServerADLS / S3KafkaIngestion utilitiesYAML configurationBronzeDatabricks / DeltaSilverdbt modelsGolddbt business logicBI reportingAnalytics

One configuration pattern, different execution paths

API utilities handle pagination and rate limits; SQL Server utilities manage incremental extraction; file utilities discover landed objects; Kafka ingestion tracks offsets and checkpoints. YAML selects known implementations rather than embedding pipeline logic or secrets.

A clear boundary between ingestion and transformation

Bronze preserves source data and ingestion metadata for traceability and replay. dbt resolves types, keys and quality rules in silver, then defines shared metrics and analytical marts in gold.

Operate the platform as a system

Version configuration and code together, coordinate dbt runs with ingestion readiness, and define ownership for schema changes, failures and backfills. BI and analytics teams consume curated models through governed access.

02 / Warehouse & application integration

Connect Reporting and Business Applications

Anonymized case study
SnowflakedbtAmazon S3AWS LambdaAPIs

The business challenge

The same data needs to support reporting and operational applications. Separate extraction scripts for every API create inconsistent business rules and make delivery failures hard to trace.

The approach

Use source-specific utility suites selected by YAML definitions to ingest APIs, SQL Server, file sources and Kafka feeds into Snowflake raw tables. dbt creates silver and gold models. BI reads curated datasets, while AWS workers or Lambda functions query approved Snowflake views and send payloads to external APIs.

Engineering scope

Ingestion configurations and utilities, Snowflake raw and curated schemas, tested dbt models, BI-ready views, AWS delivery workers, API mappings and recovery procedures.

How the data moves

Connect Reporting and Business Applications: source-to-consumption architectureSource systems: APIs · SQL Server · ADLS / S3 · Kafka. YAML + utilities: Source-specific ingestion into Snowflake. Raw → Curated: dbt builds silver and gold models. Two serving paths: BI / analytics or AWS workers / Lambda. API delivery: Workers query Snowflake and push payloadsAPIsSQL ServerADLS / S3KafkaIngestion utilitiesYAML configurationRaw tablesSnowflakeSilver / Golddbt / SnowflakeBI / AnalyticsAWS workersor LambdaAPIsPayload deliveryConnect Reporting and Business Applications: source-to-consumption architectureSource systems: APIs · SQL Server · ADLS / S3 · Kafka. YAML + utilities: Source-specific ingestion into Snowflake. Raw → Curated: dbt builds silver and gold models. Two serving paths: BI / analytics or AWS workers / Lambda. API delivery: Workers query Snowflake and push payloadsAPIsSQL ServerADLS / S3KafkaIngestion utilitiesYAML configurationRaw tablesSnowflakeSilver / Golddbt / SnowflakeBI / AnalyticsAWS workersor LambdaAPIsPayload delivery

Shared models for reporting and applications

Keep business definitions in dbt-managed Snowflake models. BI tools consume analytical marts; integration workers read dedicated views that expose only the fields required by each destination API.

Bounded, recoverable API delivery

Use Lambda for bounded tasks and longer-running AWS workers where execution time or volume requires it. Page Snowflake reads, respect destination rate limits and track delivery checkpoints, retries and idempotency keys.

Explicit access and delivery ownership

Give each worker narrowly scoped Snowflake access, keep credentials in a secrets store and record delivery outcomes. Separate extraction progress from API acknowledgements so failed requests can be retried without silently skipping data.

03 / Platform modernization

Replace Manual File Uploads with Automated Workflows

Anonymized case study
Azure Data FactoryADLSAmazon S3DatabricksSnowflake

The business challenge

Operational data lives on local computers and shared folders. Manual uploads and desktop scripts make data availability dependent on individual machines and obscure which files reached the platform.

The approach

Use Azure Data Factory with a self-hosted integration runtime to read approved local files or network shares and copy them to ADLS. For an S3 destination, use an AWS-compatible transfer utility or worker. Once landing is complete, an event or readiness check triggers ingestion into the target data platform.

Engineering scope

Source inventory, transfer and landing design, ADF pipelines or AWS transfer utilities, ingestion triggers, reconciliation checks, curated platform models and a phased cutover runbook.

How the data moves

Replace Manual File Uploads with Automated Workflows: source-to-consumption architectureLocal sources: Approved computer folders and file shares. Managed transfer: ADF + self-hosted runtime, or AWS transfer utility. Cloud landing: ADLS on Azure, or Amazon S3 on AWS. Ingestion trigger: Completed-file events or readiness checks. Data platform: Databricks / Snowflake → curated modelsLocal filesFolders / sharesAzure Data FactorySelf-hosted runtimeAWS transferUtility / workerADLSAzure landingAmazon S3AWS landingIngestion triggerEvent / readiness checkData platformDatabricks / SnowflakeReplace Manual File Uploads with Automated Workflows: source-to-consumption architectureLocal sources: Approved computer folders and file shares. Managed transfer: ADF + self-hosted runtime, or AWS transfer utility. Cloud landing: ADLS on Azure, or Amazon S3 on AWS. Ingestion trigger: Completed-file events or readiness checks. Data platform: Databricks / Snowflake → curated modelsLocal filesFolders / sharesAzure Data FactorySelf-hosted runtimeAWS transferUtility / workerADLSAzure landingAmazon S3AWS landingIngestion triggerEvent / readiness checkData platformDatabricks / Snowflake

Choose a supported transfer path

The Azure path uses ADF and a self-hosted runtime with access to the source folders, landing files in ADLS. The AWS path uses a transfer utility or worker to write to S3; it does not assume ADF has a native S3 copy destination.

Trigger only when data is ready

Ingestion starts after a file or batch is complete. Manifests or completion markers, file identity checks and repeatable ingestion prevent partial uploads and duplicate notifications from producing inconsistent tables.

Move in stages with reconciliation

Inventory source dependencies, confirm machine availability and permissions, and compare file counts and control totals before cutover. Land raw data first, then apply platform transformations and retire manual steps in a controlled sequence.

A clearer way forward

What is slowing your team down?
Let us help you find a way forward.

Tell us about the manual work, disconnected systems or unreliable data you want to improve.

Talk to us