Big Data

A Data Mesh for data-heavy-product companies. — Finding a good technical architecture for a data mesh is hard and a very company-specific process. Kafka-based data meshes are a great choice for companies that build data-heavy products already. If your software engineers won’t be comfortable processing heavy amounts of data, and are not able to get into the…

Analyzing and comparing 7 ways of deduplicating rows in a BigQuery table. — TL;DR: Use the QUALIFY clause and go to the conclusion. In this short article, we will dive into some of the different ways to deduplicate rows in a BigQuery table. Summary Introduction Replicability Method 1: Deleting rows in-place Method 2: Selecting distinct Method 3: Grouping by all keys Technical or functional deduplication? Method 4: Grouping by primary…

What is what and how do they differ? — All three architectures are now established in enterprises and are part of a solid data platform, but how do they differ from one another? However, one can say that these concepts are not only competitors but rather build and depend on one another. The Data Warehouse vs. The Data Lake Here, I will only briefly define these…