Most data teams share a familiar set of constraints: strong SQL skills but limited time for deep custom engineering, tight deadlines, and a growing sprawl of one-off pipelines that become harder to maintain the more of them exist. This is the problem DRIFT Accelerators was built to solve, a configuration-driven ETL (Extract, Transform, Load) framework developed in-house at Bajaj Tech.AI to make building and maintaining data pipelines faster, cheaper, and more consistent.

This is Part 1 of a two-part series. Here, we walk through why we built DRIFT instead of adopting existing tools, how it's designed, and what the first version focused on delivering.

The real cost of most data pipeline sprawl isn't building the first pipeline, it's maintaining the fiftieth one, each written slightly differently by whoever built it.

What Problems Was DRIFT Built to Solve?

Our journey began with a clear understanding of the challenges data teams actually face day to day challenges that most off-the-shelf tooling addresses only partially:

• SQL proficiency: Most team members are proficient in SQL, so a solution needed to lean on that existing skill set rather than requiring everyone to learn a new programming paradigm.
• Tight deadlines: In a fast-paced environment, meeting deadlines is a constant pressure that any tooling decision needs to account for.
• Centralized job management: Simplifying job creation and maintenance across the organization, rather than leaving each team to build pipelines its own way.
• Quick turnaround time (TAT): Rapid deployment of new data pipelines is essential when the business needs a new data source integrated quickly.
• Reduced maintenance overhead: Moving away from a traditional single-code, single-job structure toward a more streamlined, reusable approach.
• Cost-effectiveness: Ensuring the solution stays efficient and cost-effective as pipeline volume grows.

Why Not Just Use Airflow or dbt?

Before developing DRIFT, we evaluated existing solutions such as Apache Airflow and dbt. Both are valuable, well-established tools, but each presented real challenges in complexity, setup time, and integration with our existing infrastructure. Apache Airflow excels at orchestration but falls short in handling business logic seamlessly on its own. dbt, though powerful, required extensive setup and posed challenges around cost control and enterprise readiness for our specific context.

Neither tool was wrong for its intended purpose, they simply weren't built for the specific combination of SQL-first teams, tight deadlines, and centralized job management we needed to solve for.

How Was DRIFT Designed?

Driven by the need for a user-friendly, scalable, and cloud-agnostic solution, we built DRIFT around several core design decisions each one a direct response to a limitation we'd hit with existing tools.

1. Configuration-driven approach: Simplifying job creation and maintenance through configuration rather than custom code for every pipeline.
2. YAML format: Chosen for its user-friendly syntax and flexibility as the primary configuration language.
3. Scalability and cost management: Ensuring the solution scales across multiple instances while managing costs effectively as usage grows.
4. Cloud agnostic: Designed to be compatible with any cloud environment, avoiding lock-in to a single provider's tooling.
5. Historical run maintenance: Incorporating a persisted store (PostgreSQL) for tracking and evaluating historical job runs over time.
6. Spark compatibility: Supporting Spark jobs for enhanced processing capabilities on larger workloads.
7. Empowering SQL users: Enabling SQL-proficient team members to create data pipelines efficiently, without needing to become full-time software engineers.
8. Versatile data sources: Supporting ODBC-supported databases, S3, SFTP, and email as targets, covering the range of sources our teams actually work with.
9. Notification system: Implementing a robust notification system for managing job success, failure, and status alerts.
10. Adhoc data requests: Facilitating adhoc data requests in CSV format for offline analysis and outbound calling use cases.

What Did Version 1 Focus On?

For the initial release of DRIFT Accelerators, we focused on delivering a core set of functionalities rather than trying to solve every use case at once:

• Metadata tables for tracking job progress and data lineage
• Support for ELT (Extract, Load, Transform) workloads and S3 as a data source
• DAG (Directed Acyclic Graph) orchestration for workflow management
• Support for adhoc emails with CSV data attachments
• Data loading onto PostgreSQL and AWS Redshift for data warehousing

What Design Considerations Shaped the Framework?

Several deliberate design choices shaped how DRIFT actually works under the hood:

1. YAML structure: A structured YAML format defines inputs, outputs, job options, and notifications consistently across every pipeline.
2. Centralized credentials: Credentials are stored centrally on AWS Secrets Manager, rather than scattered across individual job configurations, for stronger security.
3. Customizable logging: Log levels can be defined at the job level, giving teams flexibility in how verbose a given pipeline's logging needs to be.
4. Grouping jobs: Job groups give teams a way to organize related pipelines together, rather than treating every job as a standalone unit.
5. Load types: The framework supports various load types, including transformations and updates, rather than assuming every job is a simple append.
6. Data modelling: Robust data modelling underpins job-related tables, including a job master, snapshot master, and job logs, giving the system a consistent internal structure to build on.

These design considerations are what separate a tool that works for a handful of pipelines from one that stays maintainable across hundreds of them.

What's Next for DRIFT?

Part 2 of this series will go deeper into the technical architecture of DRIFT Accelerators Version 1 and highlight the new features and enhancements introduced in DRIFT Version 2 exploring how Bajaj Tech.AI continues to evolve its approach to data integration and pipeline technology.

Key Takeaways

• DRIFT was built specifically around SQL-proficient teams, rather than requiring a shift to a new programming paradigm
• A configuration-driven, YAML-based approach turns new pipeline creation into a config change rather than new code
• Centralizing credentials, logging, and job metadata is what keeps a growing pipeline library maintainable over time
• Cloud-agnostic design avoids locking a data platform into a single provider's proprietary tooling
• DRIFT powers the Data Lakehouse architecture behind several of Bajaj Tech.AI's financial services data platforms

Looking to reduce the maintenance overhead of your organization's data pipelines? Connect with our experts to explore the right approach for your team.