Posts

Fabric Mirroring is Changing the Game for Data Ingestion

,
Microsoft Fabric Mirroring changes the game with Data Ingestion giving you near real-time data with a no-code framework

Microsoft Fabric Mirroring changes the game with Data Ingestion, giving you near real-time data with a no-code framework.

Microsoft’s Fabric Mirroring will change how you perform data ingestion. If you are using products to automate batch processes for data dumping, did you know that Fabric Mirroring might remove the need for these tools and provide you with near real-time access to the data as it changes in the source systems?

Suppose you have not yet heard of the medallion architecture. In that case, it involves utilizing bronze, silver, and gold layers to describe the data processing processes from intake into your data hub to consumption from your reporting applications of choice. This multi-layered approach existed before I started my analytics career in the early 2000s. Think of it simply as bronze being your unprocessed data, silver being somewhat cleaned and organized data processed from your bronze layer, and gold being your aggregated and optimized data ready for prime-time business insights.

It’s essential to understand the evolution of data management. From the ’90s to the early 2000s, the process of getting data from each application (referred to as a spoke) into your data repository (data hub) was complex. In the Microsoft world, multiple SSIS packages or other processes were used to pull data into tables with varchar(max); this was typically a batch process that ran on a schedule, leading to potential issues.  There were so many SSIS packages that we needed an automation language to build them all, rather than doing them individually.

Many businesses’ analytics projects struggle to quickly integrate the correct data into their hub so that data transformations and validations can be effective. If you get this wrong, there is no point in collecting $200 and passing Go. Your data analytics project might end up going straight to jail.

How can we load data quickly and successfully?

I am introducing you to a no-code, near-real-time option for loading your data into your data lake (data hub) within Fabric. This new feature is known as Fabric Mirroring.

While I love the functionality of Fabric Mirroring, I am not a fan of the name. Many people with SQL Server experience think this is similar to Database Mirroring because these names are similar.

In my opinion, Fabric mirroring is similar to implementing Change Data Capture (CDC) on your SQL Server databases. CDC feeds data into a real-time streaming tool like Apache Kafka to copy data from your spoke (SQL Server application database) into your hub (Data Lake).

The benefit here is twofold. First, you don’t have to manage the Change Data Capture or Kafka implementations. Second, and most importantly, this is more than just an SQL Server solution. In the future, you can use Fabric Mirroring to ingest data from all your sources (spokes) into your data hub in near real-time, with minimal to no code required.

For example, here is how to use Fabric Mirroring to import Dynamics 365 or Power Apps data into Fabric. You can do the same for Azure Cosmos Database and Snowflake. SQL Server is coming soon.

Currently, the following databases are available:

Platform Near real-time replication Type of mirroring
Microsoft Fabric mirrored databases from Azure Cosmos DB (preview) Yes Database mirroring
Microsoft Fabric mirrored databases from Azure Databricks (preview) Yes Metadata mirroring
Microsoft Fabric mirrored databases from Azure Database for PostgreSQL flexible server (preview) Yes Database mirroring
Microsoft Fabric mirrored databases from Azure SQL Database Yes Database mirroring
Microsoft Fabric mirrored databases from Azure SQL Managed Instance (preview) Yes Database mirroring
Microsoft Fabric mirrored databases from Snowflake Yes Database mirroring
Microsoft Fabric mirrored databases from SQL Server (preview) Yes Database mirroring
Open mirrored databases Yes Open mirroring
Microsoft Fabric mirrored databases from Fabric SQL database (preview) Yes Database mirroring

Now I know I can use Fabric Mirroring to help me get near real-time data into my hub with no code required. Why else should Fabric Mirroring be a game-changer for my analytics projects?

The Fabric Mirror enables us to accomplish a lot more in less time.

Suppose you have an SLA for getting data into a data warehouse in 24 hours. Processing through all the layers took you 20 hours (12 hours into bronze, 6 hours from bronze to silver, and 6 hours from silver to gold). If you now had near real-time, say 90 seconds, to get changes into bronze, that gives you an extra 11 hours and 59 minutes to improve data quality, data validation, and other processes upstream.

Centralized Data Management

Having a single hub that the applications (spokes) automatically send data to, a centralized database, and the clients and tools used, eliminates the need to install additional software. You now transition from pulling data from the spokes with batch processing to pushing data from the spokes in near real-time. It also simplifies data governance and enhances security because combining this with Preview lets you see which spokes the data goes into.

For example, you must comply with GDPR, and Sarah in the UK has now requested that her data be removed. You can now easily find the data in the spokes from the hub to determine what data needs to be purged quickly.

Simplified Data Ingestion.

Instead of mixing and matching different data sources, your delta tables will be created across your Cosmos Databases, Azure SQL databases, Dynamics 365, and other future fabric mirroring sources. You no longer need to worry about which sources are in Excel, CSV, flat file, JSON, etc. They are all in the same format, ready for you to do your transformations, data validation, and apply any business rules required for your silver level.

Improved Query Performance

Those who know me know that I love discussing query performance tuning. I am passionate about making databases go just as fast as your favorite F1 race car. I also know that you have at least one group of people running reporting queries against your line-of-business application database or an availability group replica. This leads to increased locks that slow down the original purpose of your application databases. These locks are now removed, and these reports can be sent against your data hub.

The mirrored data is also stored in an analytics-ready format, such as delta tables, which enhances query performance across various tools within Microsoft Fabric, including Power BI.

What if you cannot use Fabric Mirroring?

The sources for Microsoft Fabric to date are limited. If I had on-premise data sources or other sources that are not ready for Fabric Mirroring, I would still encourage this architecture approach of using change data capture, where available, to lead to streaming your data into your data hub of choice.

About ProcureSQL

ProcureSQL is the industry leader in providing data architecture as a service, enabling companies to harness their data and grow their business. ProcureSQL is 100% onshore in the United States and supports the four quadrants of data, including application modernization, database management, data analytics, and data visualization. ProcureSQL serves as a guide, mentor, leader, and implementer, providing innovative solutions to drive better business outcomes for all businesses. Click here to learn more about our service offerings.

/0 Comments/by

The CIO’s Guide to Leveraging Microsoft Fabric for Competitive Advantage

,

I attended the Microsoft Fabric conference for the first time last week. I wanted to provide a guide that CIOs and CEO’s could leverage to understand how they could utilize these new announcements at the 2025 Fabric Conference to obtain a competitive advantage. To be transparent, I was skeptical because Microsoft consistently changes or rebrands its analytics platform every three to five years. We have gone from Parallel Data Warehouses (PDW) to Analytics Platform Services (APS), Azure Services, Azure SQL Data Warehouse, and Azure Synapse Analytics, bringing us to Microsoft Fabric.

John Sterrett from ProcureSQL attend the 2025 Microsoft Fabric Conference

John Sterrett from ProcureSQL attends the 2025 Microsoft Fabric Conference.

To my surprise, after this conference, I have gone from seeing Fabric as Microsoft’s current take on Analytics to how it will stand out as an analytics platform of choice for people who want a simple, quick, and easy way to do analytics with the tools they already love using.

Artificial Intelligence (AI) will only be as practical as the quality of your data. Garbage in still equals garbage out, or as I like to call it, building a trusted dumpster fire. Preparing your data for AI will be the key to success with your AI Projects. Microsoft clearly understands this by focusing on preparing your data for AI with fabric mirroring, fabric databases, and SQL Server 2025. My takeaway is that you won’t have to get ready if you stay ready.

Copilot for all Fabric SKUs

Microsoft is committed to giving more people access to its AI tools as a commitment to this. In the coming weeks, users on F2 fabric compute and above can utilize Copilot. Additionally, you can use Fabric Copilot capacity, a new feature that simplifies setup, user management, and access to Copilot across different tiers.

Why Fabric Mirroring Is A Game Changer

Those following us aren’t new to the concept and advantages of fabric mirroring. One of the biggest mistakes we see that multiplies the odds of your analytics projects failing is incorrectly landing your data into your analytics platform of choice. Either the data is missing, has been transformed incorrectly, or is no longer being received.

Microsoft provides a feature called “mirroring” to help solve the problem of getting your data into your landing zone. With Azure SQL Databases and fabric databases, it’s as easy as a few clicks. Coming soon, you will have similar experiences with PostgreSQL in Azure, Oracle, SQL Server in VMs, and on-premises. What about other apps/data stores? Open mirroring is coming soon, and you can leverage it to get your other data into the Fabric landing zone.

Multi-Cloud Shortcuts

Microsoft has partnered with Snowflake to provide iceberg-formatted data across Fabric, eliminating data movement and duplication. You can use a shortcut to point directly to an Iceberg table written using Snowflake in Azure. Snowflake has also added the ability to write Iceberg tables directly into OneLake.

Apache Iceberg tables can be used with Fabric due to a feature called metadata virtualization. Behind the scenes, this feature utilizes Apache XTable.

The key takeaway is that users can now work on the same data using both Snowflake and Fabric, without requiring data movement or duplication. Letting your data professionals utilize the tools they use best is a huge win.

Fabric Databases

Microsoft Fabric Databases is the new kid on the block, and it’s already seeing traction as the first fully SaaS-ifyed database offering. Fabric databases are built for ease of use as a unified data platform. You can create databases in just a few clicks and have zero maintenance to worry about, as Microsoft fully manages the databases. Fabric database data is automatically mirrored into OneLake for analytics.

The key takeaway is that you can utilize Microsoft Fabric for application development and eliminate the need for a database infrastructure as a service MSP/partner. You can eliminate this cost as you should always get exponential value from your data MSP (what we built our practice focusing on), not just body for monitoring or keeping the lights up and running.

SQL Server 2025

Microsoft announced some updates to SQL Server 2025 at the keynote and in other breakout sessions. While it is still in private preview, it was easy to see how anyone who could write T-SQL could leverage models and vectors without needing extensive knowledge of vectors or algorithms. GraphQL will enable developers to access API endpoints and consume data, similar to most other APIs. JSON will be treated as a first-class citizen, with its data type and indexes, to help developers access their JSON data quickly and easily.

With SQL Server 2025, you can mirror your data to Microsoft Fabric with Zero ETL, zero code, our OneLake, and near real-time mirroring at no additional cost, without requiring change data capture. This will help reduce your total cost of ownership. There will be no additional compute costs for Availability Groups; continue to utilize your Fabric compute.

The key takeaway is that Microsoft continues investing in making SQL

Server more accessible from the ground to the cloud. SQL Server will continue to make it easier to help you utilize your data inside and outside the relational platform.

Other notable features

Autoscale Billing for Spark optimizes Spark job costs by offloading your data’s extraction, load, and transformation to a serverless billing model.

Command-line interface Fabric CLI is now in preview. Built on fabric APIs, it is designed for automation. There will be less clicky-clicky and more scripts that you can version control.

API and Terraform Integration Automate key aspects of your fabric platform now by utilizing Terraform. If you have used it with Azure, get ready to use it with Fabric as well.

CI/CD enhancements. With Fabric’s git integration, multiple developers can frequently make incremental workspace updates. You could also utilize variable libraries and delivery pipelines to help get your changes vetted and tested quickly through your various testing environments.

User Data Functions Fabric user data functions is a platform that allows you to host and run applications on Fabric. Data engineers can write custom business logic and embed it into the fabric ecosystem.

Statistics That Caught My Attention

  • Microsoft Fabric supports over 19,000 organizations, including 74% of Fortune 500 companies.
  • Power BI has over 275k users, including 95% of Fortune 500 companies
  • 45k consultants trained, 23k partner certifications in its first year
  • One billion new apps will be built in the next five years.
  • 87% of leaders believe AI will give their organization a competitive edge
  • 30,000+ fabric certifications completed in twelve months

I will be back next year and will provide you with another write-up, similar to the one I produced this week, in case you are unable to attend.

About ProcureSQL

ProcureSQL is the industry leader in providing data architecture as a service, enabling companies to harness their data and grow their business. ProcureSQL is 100% onshore in the United States and supports the four quadrants of data, including application modernization, database management, data analytics, and data visualization. ProcureSQL works as a guide, mentor, leader, and implementer to provide innovative solutions to drive better business outcomes for all businesses. Click here to learn more about our service offerings.

Do you have questions about leveraging AI, Microsoft Fabric, or the Microsoft Data Platform? You can chat with me for free one-on-one, or contact the team. We would love to share our knowledge and experience with you.

/0 Comments/by