September Homeruns 🏁⚾

Power BI

What is it: Power BI has joined the dark side... with a dark mode! Let us know if you’re embracing the dark side too in the comments section below (yes, we have that)!

The monthly release for Power BI is out, and while dark mode might be stealing the spotlight, there’s another major update you won’t want to miss—now you can edit direct lake mode models right in Power BI Desktop. This means you can add new measures, update table schemas, and more, all without needing to download the dataset file to your desktop. For enterprise deployment, there’s Git integration available by exporting a project (PBIP) file, which enhances version control.

Why it matters: Live editing a dataset in Power BI service is a convenient feature, especially when the model is too complex to download and run on your local machine. Fabric will utilize its own capacity to execute editing commands, so you won’t have to stress about resource management. That said, it’s crucial to have governance mechanisms in place to prevent capacity overutilization and keep costs in check.

All about the new features here.

Sigma Computing

What is it: If you haven’t heard about Sigma yet, it’s a BI tool designed to give you a Microsoft Excel-like experience for analyzing data. You can query data warehouses directly and calculate metrics or create columns using Excel-like formulas.

What’s new: In their latest release, Sigma has rolled out a bunch of features to better compete with others in the market. Here are some highlights you’ll definitely want to check out:

• Ability to create datasets i.e. package tables, relationships and metrics in a single reusable object which can then be shared widely across an organization

  

  Source: Sigma Computing

• “Explain viz” feature that generates AI summarized descriptions of visuals on a canvas

  

  Source: Sigma Computing

• Formula assistant that leverages Natural Language Processing to help translate a business question to a calculation formula

  

  Source: Sigma Computing

• And something we haven’t seen before—a unique capability that allows you to write data from the tool’s spreadsheet interface directly into Snowflake and Databricks. This update includes secure write capability using OAuth (Open Authentication), enabling Single Sign-On.

Some other cool features include - 1) Glean integration, to help connect the dots between data analysis and unstructured company data (chats, logs etc.), 2) Overlay containers to provide additional context about the dashboard, and 3) an upcoming SQL editor.

Why it matters: Sigma is tackling the challenge of democratizing self-service analytics by designing features that feel familiar to Excel users. This could lower barriers for organizations aiming to empower a wider user base with data. However, with options like dataset creation and write-back capabilities to data warehouses, it’s crucial to implement strong governance models to prevent compute costs and data assets from spiraling out of control.

All about the new features here

Microsoft Fabric

What is it: Microsoft Fabric has just introduced mirroring for Azure Databricks Unity Catalog, and it’s a pretty nifty addition! This catalog is essentially a one-stop shop for schemas, tables, views, orchestration jobs, and more, all wrapped up in a neat package. It offers granular access controls, making governance a breeze. With mirroring, you can say goodbye to the hassle of copying data over to Fabric, which should help save on storage costs and reduce the risk of pesky pipeline failures.

Why it matters: Mirroring a Unity Catalog is great news if you’re keen on building your data warehouses in Fabric. It’s a smart move if you want to keep your powerful Databricks compute free for complex transformations or data science tasks. That said, don’t forget to weigh the costs of setting up your warehouse in Databricks against using direct mode in Power BI—it could lead you to some similar outcomes without breaking the bank.

Check it out here.

Databricks

What is it: Databricks has just launched a fresh new interface for its workflows, and it’s pretty slick! Now, you can view job runs projected on a timeline, broken down by tasks and dependencies. This visual representation makes it easier to spot the most time-consuming tasks in your jobs and tackle potential issues before they become major headaches.

Additionally, there are updates like run events to capture details about job runs and simplified error codes that make it easier to understand any problems that pop up. However, the real standout feature is the integration of the Databricks Assistant with workflows to help troubleshoot job failures. This context-aware feature is currently available only with Notebooks, but it’s definitely worth checking out!

The Databricks Assistant is your new best friend when it comes to fixing those pesky recurring errors in your code—whether it’s incorrect tables, functions, syntax errors, or those annoying trailing commas. Say goodbye to hours of hair-pulling over a missed comma! Check it here.

Why it matters: Optimizing the efforts of data engineers is crucial when working with data. It not only helps save costs but also reduces the risk of downtime. The features introduced by Databricks are designed to boost developer productivity and streamline error resolution in pipelines. This means data engineers can spend less time troubleshooting and more time tackling innovative and impactful use cases.

Read more here.

Microsoft

What is it: Incremental refresh has officially landed in Preview for Dataflow Gen2! This new feature lets you configure Dataflow to fetch and refresh only the data that has changed since your last refresh. Talk about saving time and resources!

Currently, incremental refresh supports destinations like Fabric Warehouse, Azure SQL Database, and Azure Synapse Analytics. It’s important to note that this feature shines best when used with sources where query folding is fully achievable—basically, when the analysis engine can translate your native query to SQL to boost performance.

Why it matters: This feature is a nice addition for when you’re dealing with large fact tables packed with years of data that don’t change often. By using incremental refresh, you can save on compute resources and costs since there’s no need to refresh entire tables. It’s all about efficiency and making your data processes smoother!

Read more here.

Data events action calendar

In case you missed it, here is the event calendar for all the upcoming data events 👇:

1️⃣ Snowflake is going on a world tour! It will be in Dallas (October 1st), Atlanta (October 3rd), NYC (October 15th), Toronto (October 21st) and Chicago (November 4th) ! Details here.
2️⃣ Data Engineer Things' virtual Data Engineering & ML Summit October 3rd-4th . Hear from Data Practitioners on real-world problem statements! Register here.
3️⃣ Dbt labs is hosting Coalesce (analytics engineering conference) 2024 in Las Vegas from October 7th-10th. The agenda includes workshops and certifications which has got us excited. Details here.
4️⃣ IBM TechXchange in Las Vegas on October 21st. Dive deep with the Business Analytics User Community on Planning, Cognos, and Controller updates! Details here.
5️⃣ The free Snowflake Ascent virtual hands on training on Snowflake Cloud is on October 28th. Details here.
6️⃣ Databricks Data + AI World Tour coming to Toronto on November 14th! Mark your calendar, this is one you can’t afford to miss! Details here.

Cool Jobs in Data!

• Affirm is looking for a remote Analyst, Growth Analytics in Canada

• Google Cloud is looking for Digital Analytics Sales Specialist in Toronto, ON

• JPMorgan is looking for a Data Owner Sr Associate (Plano, TX)

• Databricks is looking for a Solution Architect (Remote, US)

Enjoying reading the latest in the data world? Please subscribe and spread the word!