Data Engineering

Benefits and costs of using Airflow in a cloud-native environment — Apache Airflow has long been a top choice for developers to manage and orchestrate complex data workflows in a structured way. Google Cloud Composer brings Airflow to the cloud and offers it in a fully-managed cloud-native environment. What advantages does this bring to developers, and what are the potential downsides? …

Writing high-performance code without creating more problems for yourself in the future. — Congratulations! 🎉 If you’re reading this, you likely want to get better at writing code for data engineering. In this guide, I will be showing you how I approach writing code as a means of problem solving. A bit about me I’m a Senior Data Engineer at Headspace Health and have been practicing data engineering for…

by Ada Sharoni, Ariel Krinitsi, Wojciech Indyk, Irach Ramos Monitoring Data Quality is becoming a frequent requirement in any data pipeline, including Spark based applications. Suppose we want to make sure our data comes in certain shape and size, and be alerted when certain criteria set is not met. Why…

With a Java UDF in Snowflake that geolocates IP addresses using MaxMind’s GeoLite2 Free Geolocation Data — including support for IPv4 and IPv6. No pre-processing needed ❄️🐻‍❄️📍 — Some backstory At my previous job I published “Geolocation with BigQuery: De-identify 76 million IP addresses in 20 second”. There I described how I transformed MaxMind’s GeoLite2 Free Geolocation database into a SQL table, ready to produce fast results thanks to a couple of “clever” SQL tricks.

A Data Mesh implementation (like every significant transformation) should survive for a minimum of 10–15 years, possibly even more. Are you planning for that? During the past months, while Data Mesh is becoming increasingly popular, I have seen many people and companies sharing their architecture to implement the Data Mesh…

Perhaps one of the hottest trend in machine learning is to generate image from a string of text, thankfully for the works presented by Stable Diffusion and DALL-E. However, training such of model is computationally expensive and require a lot of data. In this article, I wanted to share the…

Introduction I was just catching on our internal architecture KT series (note to self: stop having so many freaking meetings so I can attend on said day) and was watching a presentation on how a team went about solidifying their pipeline engineering with best practices, and it made me think about…

So often with tech solutions the right answer is a point in time and space around which requirements, technologies and capabilities swirl (tenuous link to the the amazing Carina Nebula captured from the JWST). Despite building our Property Data API (Kraken) only 18 months ago (see our blog post from…

Written by Ylva Lundegård and Dhiana Deva There are currently hundreds of millions of companies worldwide, and thousands of new companies are founded daily. It is extremely challenging to track and analyze them all, but EQT has a secret weapon, Motherbrain. Let’s take a look at the data architecture behind…