Two Solutions for Building Modern Data Architectures
Dr. André Kaderli is a Senior Data Scientist and part of the expert team at Novalytica. In this role, he is responsible for designing complex data architectures tailored to the specific needs of an organisation. A comprehensive data architecture enables companies to leverage data from disparate systems and automate data flows—for instance, providing regular reporting for various stakeholders. The first step involves understanding the existing system landscape and the requirements regarding data usage, as well as defining the future setup. In this context, André Kaderli frequently encounters the question of whether Microsoft Fabric or Databricks is the more suitable solution. In the following interview, he answers the most critical questions.
André, can you briefly explain the context in which Databricks and Microsoft Fabric are used and what their fundamental differences are?
Both solutions serve to build modern Lakehouse architectures, which combine the strengths of Data Lakes and Data Warehouses to efficiently store, process, manage, and analyse large volumes of data.
Databricks is an established, developer-oriented platform based on an open Data Lakehouse model using Delta Lake. It offers high flexibility and scalability, supporting various cloud providers such as Microsoft Azure, AWS, and Google Cloud. Databricks is particularly suited for complex, code-based data processes and advanced analytics, including Machine Learning and AI.
Microsoft Fabric, on the other hand, is a younger, fully integrated SaaS approach from Microsoft, launched in late 2023. It is based on the central storage concept OneLake and also utilises the Delta format. Fabric is heavily focused on user-friendliness and Low-Code/No-Code usage, with a user interface aligned with Power BI. This facilitates entry for self-service teams but offers fewer individual configuration options.
Under which circumstances do you recommend Fabric or Databricks to organisations?
Databricks is ideal for companies seeking a highly configurable and scalable solution for complex data processing and advanced analytics, particularly through code-based workflows. The platform integrates seamlessly with services like Azure Data Factory and Power BI and, if configured efficiently, can be operated cost-effectively even for smaller applications.
Microsoft Fabric is primarily aimed at organisations already deeply rooted in the Microsoft ecosystem that focus on simple integration, rapid results, and Self-Service BI. The platform supports Low-Code and No-Code solutions for data integration and transformation, providing a UI that helps business users get started quickly.
How can these solutions be integrated into existing IT landscapes, and what challenges does implementation pose?
Databricks offers high technological flexibility and can be combined with various cloud providers and on-premise systems. It supports both structured and unstructured data. Using open standards like Apache Spark and Delta Lake facilitates integration but requires technical expertise.
Microsoft Fabric scores with its easy integration into existing Microsoft environments. For companies heavily reliant on Microsoft technologies, the seamless embedding of Fabric offers clear advantages. However, compared to Databricks, Fabric offers less room for technical configuration, which can be a limitation for highly customised requirements.
What does Data Management and Data Governance look like in both solutions?
Databricks provides a comprehensive, integrated solution for Data Governance with Unity Catalog. It covers all data assets and enables central access control, security management, and detailed data lineage.
Microsoft Fabric utilises Microsoft Purview for governance. While Purview offers central metadata management and classification, its integration into Fabric is currently not yet fully mature, particularly regarding end-to-end lineage visualisation.
How is data prepared from the respective architectures for Power BI reports for end users?
Databricks relies on code-based data processing with Python, SQL, Scala, or R. This requires technical know-how but allows for maximum control. In practice, a Medallion Architecture is often used, where data is cleaned and transformed in stages. The final data is then integrated via Power BI connectors.
Microsoft Fabric follows a Low-Code/No-Code approach with integrated tools like Dataflows Gen2 and Pipelines. This makes it easier for business and Power BI users to enter the world of data processing. For more complex applications, code-based Notebooks are also available. Through native integration, structured data can be visualised directly without additional pipelines.
How do costs and operational efficiency differ between the two?
Costs vary by scenario. Databricks offers a flexible, consumption-based pricing model. In pay-as-you-go mode, it features automatic stopping of inactive clusters, ensuring costs are only incurred during actual use.
Microsoft Fabric uses a capacity-based pricing model, providing predictability for steady workloads. However, Fabric capacities must be stopped manually or via API, which deactivates all content in the workspace—including Power BI reports. This often makes Fabric less practical in a consumption model and more attractive for reserved capacities.
Where do you see the future development of Fabric and Databricks, and which trends should companies monitor?
Databricks is increasingly positioning itself as an end-to-end platform for Data, Analytics, and AI. It is becoming attractive for companies that want to deeply integrate AI development into an open, highly configurable infrastructure.
Microsoft Fabric aims to be a unified data platform within the Microsoft ecosystem, bringing Power BI, Data Factory, and Synapse under one roof. It targets organisations seeking close alignment between data, reporting, and operational processes.
The choice of platform should be based on long-term technical strategy, organisational type, and internal expertise. We recommend actively monitoring trends and remaining strategically flexible—seeking cross-technology consulting when necessary.
