保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Programming

Java and Data Engineering

Java Juggernaut: The key to data engineering mastery Naser Tamimi November 11, 20235 min read

DATA ENGINEERING

Photo by Zhen H on Unsplash Photo by Zhen H on Unsplash Learn this step by step with the interactive Computer Science and Data Engineer roadmaps.

Data Engineering and Programming Skills

When we think about data engineering, the first programming skills that usually come to mind are SQL and maybe Python. SQL is this well-known language for querying data, deeply ingrained in the world of data and pipelines. Python, on the other hand, has become quite powerful in data science and is now making its mark in the evolving field of data engineering. But, is this common belief accurate? Are SQL and Python really the most important programming skills for Data Engineers? In this article, I'll share my experiences on this topic, aiming to help young professionals figure out the best skills to make the most of their time and energy.

Why Java and Scala?

In today's data engineering, we handle a massive amount of data. The main job is figuring out how to gather, change, and store this huge load of data every day, hour, or even in real-time. What makes it trickier is making sure different data services can smoothly run on various systems without worrying about what's happening underneath. In the last 15 years, smart folks have come up with distributed computing frameworks to deal with this data overload. Hadoop and Spark are two big names in this game. Because both these frameworks are mainly built using JVM (Java Virtual Machine) languages (Hadoop uses Java, and Spark uses Scala), many data and software experts believe that Java and Scala are the way forward in data engineering. Moreover, the ability of JVM applications to be portable makes them an excellent choice for data applications operating across diverse systems and environments. You can develop data pipelines that seamlessly run on various cloud and local setups, allowing you to scale your systems up or down without concerns about the underlying infrastructure.

How does a data pipeline look like in a JVM-based applications?

Now that we've explored the benefits of Java and Scala, or more broadly, JVM-based data applications, in handling big data, the next logical question is: what do these applications, or simply data pipelines, look like? This section aims to provide an overview of the architecture of such applications. To begin, it's essential to develop a data pipeline in Java or Scala. Typically, multiple related data pipelines can coexist within the same Java or Scala project. For effective project management, tools like Apache Maven can be employed. Maven simplifies the creation, management, and building of Java applications, making the process more efficient and reliable. In these projects, a data pipeline often comprises one or more Java or Scala classes. Spark is commonly integrated into these classes for tasks such as reading (or extracting), transforming, and writing (or loading) data. While data can be read from and written to various sources, Hive tables are often the natural choices. Standard transformations are encapsulated in common classes, making them reusable across different pipelines. This code shows a basic data pipeline in Spark Scala. java

import org.apache.spark.sql.{SparkSession, DataFrame}
import org.apache.spark.sql.functions._

class MyExampleDataPipeline {

  val spark: SparkSession = SparkSession.builder
    .appName("DataFrame Transformation Example")
    .master("local[*]")
    .enableHiveSupport()
    .getOrCreate()

  def main(inputTableName: String, outputTableName: String): Unit = {
    val inputDataFrame: DataFrame = spark.table(inputTableName)

    val transformedDataFrame: DataFrame = inputDataFrame.SOME_TRANSFORMATIONS

    transformedDataFrame.write.mode("overwrite").format("parquet").saveAsTable(outputTableName)
  }
}

Ultimately, the goal is to build a Java application typically in the form of a jar file. This jar file, along with appropriate arguments, can be invoked using job and workflow management systems like Apache Airflow. This enables the execution of specific data pipelines at scheduled intervals, contributing to an organized and automated data processing workflow. Here's a simple example demonstrating how to execute a data pipeline, represented as a class, from the command line. bash

spark-submit --class MyExampleDataPipeline --master local[*] yourJarFile.jar your_input_hive_table_name your_output_hive_table_name

More advanced practices

As previously mentioned, data pipelines, now encapsulated in jar files, typically require scheduled execution, especially for batch processing, or activation based on triggered events, commonly for real-time processing. Apache Airflow serves as a robust solution for orchestrating these jar files and task classes, facilitating the execution of jobs on a regular schedule. Alternatively, jar files can be triggered using tools like AWS Lambda for irregular schedules and real-time processing. This is an example of an Apache Airflow DAG designed to execute a Java class daily. python

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator

jar_file_path = "/path/to/yourJarFile.jar"

input_table_name = "your_input_hive_table_name"
output_table_name = "your_output_hive_table_name"

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2023, 1, 1),
    'depends_on_past': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'my_data_pipeline_dag',
    default_args=default_args,
    description='DAG to run the MyExampleDataPipeline class',
    schedule_interval='@daily',  # Adjust the schedule as needed
)

start_task = DummyOperator(task_id='start', dag=dag)
end_task = DummyOperator(task_id='end', dag=dag)

spark_submit_command = f"spark-submit --class MyExampleDataPipeline --master local[*] {jar_file_path} {input_table_name} {output_table_name}"

run_data_pipeline_task = BashOperator(
    task_id='run_data_pipeline',
    bash_command=spark_submit_command,
    dag=dag,
)

start_task >> run_data_pipeline_task >> end_task

Moreover, Continuous Integration/Continuous Deployment (CI/CD) tools, including Jenkins, GitHub Actions, Spinnaker, among others, offer a seamless way to develop and deploy pipelines across various environments from development to testing and production environments. This ensures a smooth and automated transition of pipelines throughout the development lifecycle.

At the end ...

We explored the evolving landscape of data engineering and the essential programming skills required in this field. While SQL and Python have traditionally been associated with data engineering, the focus is shifting towards Java and Scala, particularly in the context of handling massive amounts of data through distributed computing frameworks like Hadoop and Spark. Again, we emphasized the importance of JVM (Java Virtual Machine) languages due to their portability, making them suitable for developing data applications that seamlessly run across diverse systems and environments. It delves into the architecture of JVM-based data applications, illustrating the development of data pipelines using Java or Scala, with Apache Maven aiding project management.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Naser Tamimi

See all from Naser Tamimi DataData EngineeringJavaProgrammingTechnology Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Creating a Data Pipeline with Spark, Google Cloud Storage and Big Query

Data Science On-premise and cloud working together to deliver a data product João Pedro March 6, 20239 min read

Becoming a Self-Taught Data Scientist

Artificial Intelligence What to do when college degree in data science just isn't an option Dario Radečić November 8, 20196 min read

Introducing Julia: An Alternative to Python and R for Data Science

Artificial Intelligence Meet the programing language of 21st century, aimed for scientific computing, data science, and machine learning. Dario Radečić January 23, 20206 min read

How To Compute Euclidean Distance in NumPy

Machine Learning Showcasing how to calculate euclidean distances in NumPy arrays Giorgos Myrianthous May 17, 20223 min read

Nine Pico PIO Wats with Rust (Part 1)

Programming Raspberry Pi programmable IO pitfalls illustrated with a musical example Carl Kadie January 30, 202517 min read

1.5 Years of Spark Knowledge in 8 Tips

Machine Learning My learnings from Databricks customer engagements Michael Berk December 24, 20238 min read

Introducing Google’s File Search Tool

AI applications The search giant fires its latest salvo against traditional RAG processing. Thomas Reid November 18, 202511 min read

The Quick and Easy Guide to Create Wordclouds in Python

Programming Create word clouds in different shapes to convey messages better. Thuwarakesh Murallie May 9, 20225 min read

Deploying Machine Learning Models with Heroku

Artificial Intelligence Don't just train, but also deploy: a step-by-step guide Avi Chawla September 10, 20228 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close