Example to read CSV file on HDFS through Python — When trying to read files from HDFS, I have been using Spark. There was one use case where I had no option but to use Python to read the file. This piece of code below does exactly the same. Method: 1 Replace these pieces of information from the below script: active_name_node_ip port …

Big Data Interview Questions on Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithm — This is part four of the data engineer interview series. If you have not already, I highly recommend you to read the introduction section of Part I to build an understanding of my career background. Data Engineer Interview Questions: Part I Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithmmedium.com

Big Data Interview Questions on Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithm — This is part three of the data engineer interview series. If you have not already, I highly recommend you to read the introduction section of Part one to build an understanding of my career background. Data Engineer Interview Questions: Part I Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithmmedium.com

Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithm — This is part two of the data engineer interview series. If you have not already, I highly recommend you to read the introduction section of Part one to build an understanding of my career background. Data Engineer Interview Questions: Part I Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithmmedium.com

Big Data Interview Questions Spark, SQL, Python, Data Modeling, and Data Warehouse, Data Structure & Algorithm — I am a data engineer with 3 years of experience. During the last 4 months, I have attended 75 interview sessions for the role of data engineering with 26 different companies. Amazon, ANZ, Apisero, Aviyel, Amagi, Busigence, BCG, BitClass, couture.ai, Fractal, Flipkart, Indeed, Healthplix, Lead School, Lumiq, Moveworks, Nagarro, Novo…

Query full dynamo DB table beyond 1MB with filters in place — In this blog, we will see examples of how to do a full query of a dynamo DB table. We will also dig into the concepts of applying filters and conditions to obtain only desirable records. Why? The question is why do we need to scan the DB if we can…

Using the boto3 prefix in Python we will extract all the keys of an s3 bucket at the subfolder level. — In this blog, we will see how to extract all the keys of an s3 bucket at the subfolder level and keys with specific extension. Along with this, we will also cover different examples with the boto3 client and resource. …

In this article, you will learn to submit a Python job on EKS cluster up and running with Fargate as a compute engine. Prerequisites Docker, Kubectl, EKS cluster up and running Assuming that you have mac os/Linux. Else, you will have to look for alternatives to install the dependencies. To install…