{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Niklas Lang",
  "title_listed": "Mastering Hadoop, Part 2: Getting Hands-On — Setting Up and Scaling Hadoop",
  "keywords": "",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/mastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop/",
  "publish_date": "2025-03-13",
  "corpus_stage": "",
  "notes": "",
  "id": "8e5aa39323e8",
  "collected_at": "2026-10-02T04:02:56",
  "title": "Mastering Hadoop, Part 2: Getting Hands-On — Setting Up and Scaling Hadoop | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\n# Mastering Hadoop, Part 2: Getting Hands-On — Setting Up and Scaling Hadoop\nUnderstanding Hadoop’s core components before installation and scaling\n[Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\nMarch 13, 202520 min read\n![Person in striped shirt plugging audio cords in](../images/000.png)\nImage from [Unsplash](https://unsplash.com/photos/grayscale-photography-of-man-in-striped-shirt-rgkKFn5Ysb0%20%0A)\nNow that we’ve explored [Hadoop’s role and relevance](https://towardsdatascience.com/mastering-hadoop-part-1-installation-configuration-and-modern-big-data-strategies/), it’s time to show you how it works under the hood and how you can start working with it. To start, we are breaking down Hadoop’s core components — HDFS for storage, MapReduce for processing, YARN for resource management, and more. Then, we’ll guide you through installing Hadoop (both locally and in the cloud) and introduce some essential commands to help you navigate and operate your first Hadoop environment.\n## Which components are part of the Hadoop architecture?\nHadoop's architecture is designed to be resilient and error-free, relying on several core components that work together. These components divide large datasets into smaller blocks, making them easier to process and distribute across a cluster of servers. This distributed approach enables efficient data processing—far more scalable than a centralized 'supercomputer.'\n![](../images/001.png)\nHadoop Components | Source: Author\nThe basic components of Hadoop are:\nLearn this step by step with the interactive [Data Engineer](https://roadmap.sh/data-engineer) roadmap.\n  * **Hadoop Common** comprises basic libraries and functionalities that are required by the other modules.\n  * The [**Hadoop Distributed File System (HDFS)**](https://databasecamp.de/en/data/apache-hdfs) ensures that data is stored on different servers and enables a particularly large bandwidth.\n  * **Hadoop YARN** takes care of resource distribution within the system and redistributes the load when individual computers reach their limits.\n  * **MapReduce** is a programming model designed to make the processing of large amounts of data particularly efficient.\n\n\nIn 2020, Hadoop Ozone, which is used as an alternative to HDFS, was added to this basic architecture. It comprises a distributed object storage system that was specially designed for big data workloads to better handle modern data requirements, especially in the cloud environment.\n### HDFS (Hadoop Distributed File System)\nLet’s dive into HDFS, the core storage system of Hadoop, designed specifically to meet the demands of big data processing. The basic principle is that files are not stored as a whole on a central server, but are divided into blocks of 128MB or 256MB in size and then distributed across different nodes in a computer cluster.\nTo ensure data integrity, each block is replicated three times across different servers. If one server fails, the system can still recover from the remaining copies. This replication makes it easy to fall back on another node in the event of a failure.\nAccording to its [documentation](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html), Hadoop pursues the following goals with the use of HDFS:\n  * **Fast recovery** from hardware failures by falling back on working components.\n  * Provision of **stream data processing**.\n  * **Big data framework** with the ability to process large data sets.\n  * **Standardized processes** with the ability to easily migrate to new hardware or software.\n\n\nApache Hadoop works according to the so-called master-slave principle. In this cluster, there is one node that takes on the role of the master. It distributes the blocks from the data set to various slave nodes and remembers which partitions it has stored on which computers. Only the references to the blocks, i.e. the metadata, are stored on the master node. If a master fails, there is a secondary name node that can take over.\nThe master within the Apache Hadoop Distributed File System is called a NameNode. The slave nodes, in turn, are the so-called DataNodes. The task of the DataNodes is to store the actual data blocks and regularly report the status to the NameNode that they are still alive. If a DataNode fails, the data blocks are replicated by other nodes to ensure sufficient fault tolerance.\nThe client saves files that are stored on the various DataNodes. In our example, these are located on racks 1 and 2. As a rule, there is only one DataNode per machine in a rack. Its primary task is to manage the data blocks in memory.\nThe NameNode, in turn, is responsible for remembering which data blocks are stored in which DataNode so that it can retrieve them on request. It also manages the files and can open, close, and, if necessary, rename them.\nFinally, the DataNodes carry out the actual read and write processes of the client. The client receives the required information from the DataNodes when a query is made. They also ensure the replication of data so that the system can be operated in a fault-tolerant manner.\n### MapReduce\nMapReduce is a programming model that supports the parallel processing of large amounts of data. It was originally developed by Google and can be divided into two phases:\n  * **Map** : In the map phase, a process is defined that can transform the input data into key-value pairs. Several mappers can then be set up to process a large amount of data simultaneously to enable faster processing.\n  * **Reduce** : The Reduce phase starts after all mappers have finished and aggregates all values that have the same key. The aggregation can involve various functions, such as the sum or the determination of the maximum value. Between the end of the Map phase and the start of the Reduce phase, the data is shuffled and sorted according to the keys.\n\n\nA classic application for the MapReduce mechanism is word counting in documents, such as the seven Harry Potter volumes in our example. The task is to count how often the words “Harry” and “Potter” occur. To do this, in the map phase, each word is split into a key-value pair with the word as the key and the number one as the value, as the word has occurred once.\nThe positive aspect of this is that this task can run in parallel and independently of each other, so that, for example, a mapper can run for each band or even for each page individually. This means that the task is parallelized and can be implemented much faster. The scaling depends only on the available computing resources and can be increased as required if the appropriate hardware is available. The output of the map phase could look like this, for example:\nmarkup\n\n```\n[(„Harry“, 1), („Potter“, 1), („Potter“, 1), („Harry“, 1), („Harry”, 1)]\n```\n\n![](../images/body-28a223b9ca3237a14153.png)\nMapReduce using the example of word counts in Harry Potter books | Source: Author\nOnce all mappers have finished their work, the reduce phase can begin. For the word count example, all key-value pairs with the keys “Harry” and “Potter” should be grouped and counted. \nThe grouping produces the following result:\nbash\n\n```\n[(„Harry“, [1,1,1]), („Potter“, [1,1])]\n```\n\nThe grouped result is then aggregated. As the words are to be counted in our example, the grouped values are added together:\nbash\n\n```\n[(„Harry“, 3), („Potter“, 2)]\n```\n\nThe advantage of this processing is that the task can be parallelized and at the same time only minimal file movement takes place. This means that even large volumes can be processed efficiently.\nAlthough many systems continue to use the MapReduce program, as used in the original Hadoop structure, more efficient frameworks, such as [Apache Spark](https://databasecamp.de/en/data/apache-sparks), have also been developed in the meantime. We will go into this in more detail later in the article.\n### YARN (Yet Another Resource Negotiator)\nYARN (Yet Another Resource Negotiator) manages the hardware resources within the cluster. It separates resource management from data processing, which allows multiple applications (such as MapReduce, Spark, and Flink) to run efficiently on the same cluster. It focuses on key functions such as:\n  * Management of performance and memory resources, such as CPU or SSD storage space.\n  * Distribution of free resources to running processes, for example, MapReduce, Spark, or Flink.\n  * Optimization and parallelization of job execution.\n\n\nSimilar to HDFS, YARN also follows a master-slave principle. The Resource Manager acts as the master and centrally monitors all resources in the entire cluster. It also allocates the available resources to the individual applications. The various node managers serve as slaves and are installed on each machine. They are responsible for the containers in which the applications run and monitor their resource consumption, such as memory space or CPU performance. These figures are fed back to the Resource Manager at regular intervals so that it can maintain an overview.\nAt a high level, a request to YARN looks like this: the client calls the Resource Manager and requests the execution of an application. This then searches for available resources in the cluster and, if possible, starts a new instance of the so-called Application Master, which initiates and monitors the execution of the application. This in turn requests the available resources from the node manager and starts the corresponding containers. The calculation can now run in parallel in the containers and is monitored by the Application Master. After successful processing, YARN releases the resources used for new jobs.\n### Hadoop common\nHadoop Common can be thought of as the foundation of the complete Hadoop ecosystem on which the main components can be built. It contains basic libraries, tools, and configuration files that can be used by all Hadoop components. The main components include:\n  * **Common libraries and utilities** : Hadoop Common provides a set of Java libraries, [APIs](https://databasecamp.de/en/data/api-en), and utilities needed to run the cluster. This includes, for example, mechanisms for communication between the nodes in the cluster or support for different serialization formats, such as Avro. Interfaces required for file management in HDFS or other file systems are also included.\n  * **Configuration management** : Hadoop is based on a large number of [XML](https://databasecamp.de/en/data/xml-file)-based configuration files, which define the main system parameters that are essential for operation. One central aspect is the network parameters required to control the machines in the cluster. In addition, the permitted storage locations for HDFs are defined here or the maximum resource sizes, such as the usable storage space, are determined.\n  * **Platform independence** : Hadoop was originally developed specifically for Linux environments. However, it can also be extended to other operating systems with the help of Hadoop Common. This includes native code support for additional environments, such as macOS or Windows.\n  * **Tools for I/O (input/output)** : A big data framework processes huge volumes of data that need to be stored and processed efficiently. The necessary building blocks for various file systems, such as TextFiles or Parquet, are therefore stored in Hadoop Common. It also contains the functionalities for the supported compression methods, which ensure that storage space is saved and processing time is optimized.\n\n\nThanks to this uniform and central code base, Hadoop Common provides improved modularity within the framework and ensures that all components can work together seamlessly.\n### Hadoop Ozone\nHadoop Ozone is a distributed object storage system that was introduced as an alternative to HDFS and was developed specifically for big data workloads. HDFS was originally designed for large files with many gigabytes or even terabytes. However, it quickly reaches its limits when a large number of small files need to be stored. The main problem is the limitation of the NameNode, which stores metadata in RAM and, therefore, encounters memory problems when billions of small files are kept.\nIn addition, HDFS is designed for classic Hadoop use within a computing cluster. However, current architectures often use a hybrid approach with storage solutions in the cloud. Hadoop Ozone solves these problems by providing a scalable and flexible storage architecture that is optimized for [Kubernetes](https://databasecamp.de/en/data/kubernetes-basics) and hybrid cloud environments.\nUnlike HDFS, where a NameNode handles all file metadata, Hadoop Ozone introduces a more flexible architecture that doesn't rely on a single centralized NameNode, improving scalability. Instead, it uses the following components: \n  * The **Ozone Manager** corresponds most closely to the HDFS NameNode, but only manages the bucket and volume metadata. It ensures efficient management of the objects and is also scalable, as not all file metadata has to be kept in RAM.\n  * The **Storage Container Manager (SCM)** can best be imagined as the DataNode in HDFS and it has the task of managing and replicating the data in so-called containers. Various replication strategies are supported, such as triple copying or erasure coding to save space.\n  * The **Ozone 3 Gateway** has an S3-compatible API so it can be used as a replacement for Amazon S3. This means that applications developed for AWS S3 can be easily connected to Ozone and interact with it without the need for code changes.\n\n\nThis structure gives Hadoop Ozone various advantages over HDFS, which we have briefly summarized in the following table:  \n| **Attribute**  | **Hadoop Ozone**  | **HDFS**  |  \n| --- | --- | --- |  \n| Storage Structure  | Object-based (buckets & keys)  | Block-based (files & blocks)  |  \n| Scalability  | Millions to billions of small files  | Problems with many small files  |  \n| NameNode - Dependency  | No central NameNode & scaling possible  | NameNode is bottleneck  |  \n| Cloud Integration  | Supports S3 API, Kubernetes, multi-cloud  | Strongly tied to the Hadoop Cluster  |  \n| Replication Strategy  | Classic 3-fold replication or erasure coding  | Only 3-fold replication  |  \n| Applications  | Big data, Kubernetes, hybrid cloud, S3 replacement  | Traditional Hadoop workloads  |  \nHadoop Ozone is a powerful extension of the ecosystem and enables the implementation of hybrid cloud architectures that would not have been possible with HDFS. It is also easy to scale as it is no longer dependent on a central name node. This means that big data applications with many, but small, files, such as those used for sensor measurements, can also be implemented without any problems.\n## How to start with Hadoop?\nHadoop is a robust and scalable big data framework that powers some of the world's largest data-driven applications. While it can seem overwhelming for beginners due to its many components, this guide will walk you through the first steps to get started with Hadoop in simple, easy-to-follow stages.\n**Installation of Hadoop**\nBefore we can start working with Hadoop, we must first install it in our respective environment. In this chapter, we differentiate between several scenarios, depending on whether the framework is installed locally or in the cloud. At the same time, it is generally advisable to work on systems that use Linux or macOS as the operating system, as additional adaptations are required for Windows. In addition, Java should already be available, at least Java 8 or 11, and internal communication via SSH should be possible.\n**Local Installation of Hadoop**\nTo try out Hadoop on a local computer and familiarize yourself with it, you can perform a single-node installation so that all the necessary components run on the same computer. Before starting the installation, you can check the latest version you want to install at <https://hadoop.apache.org/releases.html>, in our case this is version 3.4.1. If a different version is required, the following commands can simply be changed so that the version number in the code is adjusted.\nWe then open a new terminal and execute the following code, which downloads the specified version from the Internet, unpacks the directory, and then changes to the unpacked directory.\nbash\n\n```\n&#119;&#103;&#101;&#116; https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz\ntar -xvzf hadoop-3.4.1.tar.gz\ncd hadoop-3.4.1\n```\n\nIf there are errors in the first line, this is most likely due to a faulty link and the version mentioned may no longer be accessible. A more up-to-date version should be used and the code executed again. The installation directory has a size of about one gigabyte.\nThe environment variables can then be created and set, which tells the system under which directory Hadoop is stored on the computer. The `PATH` variable then allows Hadoop commands to be executed from anywhere in the terminal without having to set the full path for the Hadoop installation.\nbash\n\n```\nexport HADOOP_HOME=~/hadoop-3.4.1 \nexport PATH=$PATH:$HADOOP_HOME/bin\n```\n\nBefore we start the system, we can change the basic configuration of Hadoop, for example, to define specific directories for HDFS or specify the replication factor. There are a total of three important configuration files that we can adjust before starting:\n  * `core-site.xml` configures basic Hadoop settings, such as the connection information for multiple nodes.\n  * `hdfs-site.xml` contains special parameters for the HDFS setup, such as the typical directories for data storage or the replication factor, which determines how many replicas of the data are stored.\n  * `yarn-site.xml` configures the YARN component, which is responsible for resource management and job scheduling.\n\n\nFor our local test, we can adjust the HDFS configuration so that the replication factor is set to 1, as we are only working on one server, and replication of the data is, therefore, not useful. To do this, we use a text editor, in our case `nano`, and open the configuration file for HDFS:\nbash\n\n```\nnano $HADOOP_HOME/etc/hadoop/hdfs-site.xml\n```\n\nThe file then opens in the terminal and probably does not yet have any entries. A new XML with the property key can then be added within the configuration area:\nbash\n\n```\n<property> \n    <name>dfs.replication</name> \n    <value>1</value> \n</property>\n```\n\nVarious properties can then be set according to this format. The different keys that can be specified in the configuration files, including the permitted values, can be found at <https://hadoop.apache.org/docs/current/hadoop-project-dist/>. For HDFS, this overview can be seen [here](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml).\nNow that the configuration has been completed, Hadoop can be started. To do this, HDFS is initialized, which is the first important step after a new installation, and the directory that is to be used as the NameNode is formatted. The next two commands then start HDFS on all nodes that are configured in the cluster and the resource management YARN is started.\nbash\n\n```\nhdfs namenode -format \nstart-dfs.sh \nstart-yarn.sh\n```\n\nProblems may occur in this step if Java has not yet been installed. However, this can easily be done with the corresponding installation. In addition, when I tried this on macOS, the NameNode and DataNode of HDFS had to be started explicitly:\nbash\n\n```\n~/hadoop-3.4.1/bin/hdfs --daemon start namenode\n~/hadoop-3.4.1/bin/hdfs --daemon start datanode\n```\n\nFor YARN, the same procedure works for the Resource and NodeManager:\nbash\n\n```\n~/hadoop-3.4.1/bin/yarn --daemon start resourcemanager\n~/hadoop-3.4.1/bin/yarn --daemon start nodemanager\n```\n\nFinally, the running processes can be checked with the _jps_ command to see whether all components have been started correctly.\n## Hadoop installation in a distributed system\nFor resilient and productive processes, Hadoop is used in a distributed environment with multiple servers, known as nodes. This ensures greater scalability and availability. A distinction is typically made between the following cluster roles:\n  * **NameNode** : This role stores the metadata and manages the file system (HDFS).\n  * **DataNode** : This is where the actual data is stored and the calculations take place.\n  * **ResourceManager & NodeManagers**: These manage the cluster resources for YARN.\n\n\nThe same commands that were explained in more detail in the last section can then be used on the individual servers. However, communication must also be established between them so that they can coordinate with each other. In general, the following sequence can be followed during installation:\n  1. Set up several Linux-based servers to be used for the cluster.\n  2. Set up SSH access between the servers so that they can communicate with each other and send data.\n  3. Install Hadoop on each server and make the desired configurations.\n  4. Assign roles and define the NameNodes and DataNodes in the cluster.\n  5. Format NameNodes and then start the cluster.\n\n\nThe specific steps and the code to be executed then depend more on the actual implementation.\n## Hadoop installation in the cloud\nMany companies use Hadoop in the cloud to avoid having to operate their own cluster, potentially save costs, and also be able to use modern hardware. The various providers already have predefined programs with which Hadoop can be used in their environments. The most common Hadoop cloud services are:\n  * **AWS EMR (Elastic MapReduce)** : This program is based on Hadoop and, as the name suggests, also uses MapReduce, which allows users to write their programs in Java that process and store large amounts of data in a distributed manner. The cluster runs on virtual servers in the Amazon Elastic Compute Cloud (EC2) and stores the data in the Amazon Simple Storage Service (S3). The keyword “Elastic” comes from the fact that the system can change dynamically to adapt to the required computing power. Finally, AWS EMR also offers the option of using other Hadoop extensions such as Apache Spark or Apache Presto.\n  * **Google Dataproc** : Google's alternative is called Dataproc and enables a fully managed and scalable Hadoop cluster in the Google Cloud. It is based on BigQuery and uses Google Cloud Storage for data storage. Many companies, such as Vodafone and Twitter are already using this system.\n  * **Azure HDInsight** : The Microsoft Azure Cloud offers HDInsight for complete Hadoop use in the cloud and also provides support for a wide range of other open-source programs.\n\n\nThe overall advantage of using the cloud is that no manual installation and maintenance work is required. Several nodes are used automatically and more are added depending on the computing requirements. For the customer, the advantage of automatic scaling is that costs can be controlled and only what is used is paid for.\nWith an on-premise cluster, on the other hand, the hardware is usually set up in such a way that it is still functional even at peak loads so that the entire hardware is not required for a large part of the time. Finally, the advantage of using the cloud is that it makes it easier to integrate other systems that run with the same provider, for example.\n## Basic Hadoop commands for beginners\nRegardless of the architecture selected, the following commands can be used to perform very general and frequently recurring actions in Hadoop. This covers all areas that are required in an ETL process in Hadoop.\n  * **Upload File to HDFS** : To be able to execute an HDFS command, the beginning `hdfs dfs` is always required. You use put to define that you want to upload a file from the local directory to HDFS. The `local_file.txt` describes the file to be uploaded. To do this, the command is either executed in the directory of the file or the complete path to the file is added instead of the file name. Finally, use `/user/hadoop/` to define the directory in HDFS in which the file is to be stored.\n\n\nbash\n\n```\nhdfs dfs -put local_file.txt /user/hadoop/\n```\n\n  * **List files in HDFS** : You can use `-ls` to list all files and folders in the HDFS directory /user/hadoop/ and have them displayed as a list in the terminal.\n\n\nbash\n\n```\nhdfs dfs -put local_file.txt /user/hadoop/\n```\n\n  * **Download file from HDFS** : The `-get` parameter downloads the file `/user/hadoop/file.txt` from the HDFS directory to the local directory. The dot _._ indicates that the file is stored in the current local directory in which the command is being executed. If this is not desired, you can define a corresponding local directory instead.\n\n\nbash\n\n```\nhdfs dfs -get /user/hadoop/file.txt \n```\n\n  * **Delete files in HDFS** : Use _-rm_ to delete the file _/user/hadoop/file.txt_ from the HDFS directory. This command also automatically deletes all replications that are distributed across the cluster.\n\n\nbash\n\n```\nhdfs dfs -rm /user/hadoop/file.txt\n```\n\n  * **Start MapReduce command (process data)** : MapReduce is the distributed computing model in Hadoop that can be used to process large amounts of data. Using `hadoop jar` indicates that a Hadoop job with a “.jar” file is to be executed. The corresponding file containing various MapReduce programs is located in the directory `/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar`. From these examples, the `wordcount` job is to be executed, which counts the words occurring in a text file. The data to be analyzed is located in the HDFS directory `/input` and the results are then to be stored in the directory `output/`.\n\n\nbash\n\n```\nhadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount input/ output/\n```\n\n  * **Monitor the progress of a job** : Despite the distributed computing power, many MapReduce jobs take a certain amount of time to run, depending on the amount of data. Their status can therefore be monitored in the terminal. The resources and running applications can be displayed using YARN. To be able to execute a command in this system, we start with the command _yarn,_ and with the help of application-list we get a list of all active applications. Various information can be read from this list, such as the unique ID of the applications, the user who started them, and the progress in %.\n\n\nbash\n\n```\nyarn application -list\n```\n\n  * **Display logs of a running job** : To be able to delve deeper into a running process and identify potential problems at an early stage, we can read out the logs. The logs command is used for this, with which the logs of a specific application can be called up. The unique application ID is utilized to define this application. To do this, the APP_ID must be replaced by the actual ID in the following command, and the greater than and less than signs must be removed.\n\n\nbash\n\n```\nyarn logs -applicationId <APP_ID>\n```\n\nWith the help of these commands, data can already be saved in HDFS, and MapReduce jobs can also be created. These are the central actions for filling the cluster with data and processing it.\n## Debugging & logging in Hadoop\nFor the cluster to be sustainable in the long term and to be able to read out errors, it is important to master basic debugging and logging commands. As Hadoop is a distributed system, errors can occur in a wide variety of components and nodes. It is therefore essential that you are familiar with the corresponding commands to quickly find and switch off errors.\nDetailed log files for the various components are stored in the `$HADOOP_HOME/logs`directory. The log files for the various servers and components can then be found in their subdirectories. The most important ones are:\n  * **NameNode-Logs** contains information about the HDFS metadata and possible connection problems:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/hadoop-hadoop-namenode-<hostname>.log \n```\n\n  * **DataNode logs** show problems with the storage of data blocks:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/hadoop-hadoop-datanode-<hostname>.log\n```\n\n  * YARN **ResourceManager logs** reveal possible resource problems or errors in job scheduling:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/yarn-hadoop-resourcemanager-<hostname>.log\n```\n\n  * **NodeManager logs** help with debugging executed jobs and their logic:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/yarn-hadoop-nodemanager-<hostname>.log\n```\n\nWith the help of these logs, specific problems in the processes can be identified and possible solutions can be derived from them. However, if there are problems in the entire cluster and you want to check the overall status across individual servers, it makes sense to carry out a detailed cluster analysis with the following command:\nbash\n\n```\nhdfs dfsadmin -report\n```\n\nThis includes the number of active and failed DataNodes, as well as the available and occupied storage capacities. The replication status of the HDFS files is also displayed here and additional runtime information about the cluster is provided. An example output could then look something like this:\nbash\n\n```\nConfigured Capacity: 10 TB\nDFS Used: 2 TB\nRemaining: 8 TB\nNumber of DataNodes: 5\nDataNodes Available: 4\nDataNodes Dead: 1\n```\n\nWith these first steps, we have learned how to set up a Hadoop in different environments, store and manage data in HDFS, execute MapReduce jobs, and read the logs to detect and fix errors. This will enable you to start your first project in Hadoop and gain experience with big data frameworks.\nIn this part, we covered the core components of Hadoop, including HDFS, YARN, and MapReduce. We also walked through the installation process, from setting up Hadoop in a local or distributed environment to configuring key files such as core-site.xml and hdfs-site.xml. Understanding these components is crucial for efficiently storing and processing large datasets across clusters.\nIf this basic setup is not enough for your use case and you want to learn how you can extend your Hadoop cluster to make it more adaptable and scalable, then our next part is just right for you. We will dive deeper into the large Hadoop ecosystem including tools like Apache Spark, HBase, Hive, and many more that can make your cluster more scalable and adaptable. Stay tuned!\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-d123f8ba604c7f334425.jpeg)\nWritten By\n## [Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\n[See all from Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Big Data](https://towardsdatascience.com/tag/big-data/)[Data Processing](https://towardsdatascience.com/tag/data-processing/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Hadoop](https://towardsdatascience.com/tag/hadoop/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F&title=Mastering%20Hadoop%2C%20Part%202%3A%20Getting%20Hands-On%20%E2%80%94%20Setting%20Up%20and%20Scaling%20Hadoop)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F&title=Mastering%20Hadoop%2C%20Part%202%3A%20Getting%20Hands-On%20%E2%80%94%20Setting%20Up%20and%20Scaling%20Hadoop)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F&text=Mastering%20Hadoop%2C%20Part%202%3A%20Getting%20Hands-On%20%E2%80%94%20Setting%20Up%20and%20Scaling%20Hadoop)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/002.jpg)](https://towardsdatascience.com/exploring-the-contents-of-dlis-files-with-python-38585157dbac/)\n## [Exploring the Contents of DLIS Files with Python](https://towardsdatascience.com/exploring-the-contents-of-dlis-files-with-python-38585157dbac/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWorking with Pandas and dlisio to Explore Well Log Data\n[Andy McDonald](https://towardsdatascience.com/author/andymcdonaldgeo/)\nJuly 14, 20238 min read\n  * [![](../images/003.png)](https://towardsdatascience.com/why-your-data-pipelines-need-closed-loop-feedback-control-76e28e3565f/)\n## [Why Your Data Pipelines Need Closed-Loop Feedback Control](https://towardsdatascience.com/why-your-data-pipelines-need-closed-loop-feedback-control-76e28e3565f/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nRealities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale\n[Jeff Chou](https://towardsdatascience.com/author/jeff.b.chou/)\nSeptember 10, 20239 min read\n  * [![](../images/004.jpg)](https://towardsdatascience.com/building-pipelines-in-apache-airflow-for-beginners-58f87a1512d5/)\n## [Building Pipelines In Apache Airflow - For Beginners](https://towardsdatascience.com/building-pipelines-in-apache-airflow-for-beginners-58f87a1512d5/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA quick and simple demo for running DAGs on Airflow\n[Aashish Nair](https://towardsdatascience.com/author/aashishnair/)\nMarch 15, 20234 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/extract-load-transform-learn-and-serve-getting-value-from-data-4f08d70bf343/)\n## [Extract & Load, Transform, Learn, and Serve - Getting Value from Data](https://towardsdatascience.com/extract-load-transform-learn-and-serve-getting-value-from-data-4f08d70bf343/)\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\nWe've all heard about ELT and ETL, but what about ELTLS?\n[Raghotham Murthy](https://towardsdatascience.com/author/raghotham/)\nJune 9, 20217 min read\n  * [![](../images/006.jpeg)](https://towardsdatascience.com/how-i-use-tkinter-python-library-to-run-algorithms-6be85f8f4e5b/)\n## [How I use Tkinter Python Library to Run Algorithms](https://towardsdatascience.com/how-i-use-tkinter-python-library-to-run-algorithms-6be85f8f4e5b/)\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\nIf you're running several python scripts for your ETL or algorithms - Tkinter is a python library that can provide GUI elements and...\n[Kathleen Lara](https://towardsdatascience.com/author/itskathleenlara/)\nMarch 10, 20223 min read\n  * [![](../images/007.jpeg)](https://towardsdatascience.com/what-i-learned-building-platforms-at-stitch-fix-fc5e0ec72c86/)\n## [What I learned building platforms at Stitch Fix](https://towardsdatascience.com/what-i-learned-building-platforms-at-stitch-fix-fc5e0ec72c86/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nFive lessons\n[Stefan Krawczyk](https://towardsdatascience.com/author/stefan.krawczyk/)\nSeptember 26, 202223 min read\n  * [![](../images/008.jpeg)](https://towardsdatascience.com/you-do-need-math-for-machine-learning-cf934a607960/)\n## [You DO need math for Machine Learning](https://towardsdatascience.com/you-do-need-math-for-machine-learning-cf934a607960/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nDon't skip the most unappreciated skill in ML. Write code AND learn the theory.\n[Sarem Seitz](https://towardsdatascience.com/author/sarem-seitz/)\nJuly 2, 202112 min read\n  * [![](../images/009.jpeg)](https://towardsdatascience.com/dbt-cli-model-selection-52ddd038d8b2/)\n## [Model Selection For dbt CLI](https://towardsdatascience.com/dbt-cli-model-selection-52ddd038d8b2/)\n[Programming](https://towardsdatascience.com/category/programming/)\nA complete cheatsheet for selecting specific models when running dbt commands\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nJanuary 25, 20238 min read\n  * [![](../images/010.jpeg)](https://towardsdatascience.com/how-to-go-from-a-python-newbie-to-a-google-certified-tensorflow-developer-under-two-months-3d1535432284/)\n## [How to go from a Python newbie to a Google Certified TensorFlow Developer under two months.](https://towardsdatascience.com/how-to-go-from-a-python-newbie-to-a-google-certified-tensorflow-developer-under-two-months-3d1535432284/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA learning journey (with tips and tricks) on how I got certified as a TensorFlow Developer by Google within two months.\n[Grady Matthias Oktavian](https://towardsdatascience.com/author/radvian/)\nSeptember 1, 202012 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "metadata": {
    "title": "Mastering Hadoop, Part 2: Getting Hands-On — Setting Up and Scaling Hadoop | Towards Data Science",
    "description": "Understanding Hadoop’s core components before installation and scaling",
    "keywords": null,
    "author": "Niklas Lang",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Mastering Hadoop, Part 2: Getting Hands-On — Setting Up and Scaling Hadoop",
    "og:description": "Understanding Hadoop’s core components before installation and scaling",
    "og:url": "https://towardsdatascience.com/mastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Scaling-Hadoop-3.png",
    "og:image:width": "1600",
    "og:image:height": "1334",
    "og:image:type": "image/png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Niklas Lang",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "20 min read",
    "article:published_time": "2025-03-13T15:42:01.000Z",
    "article:modified_time": "2025-03-13T15:42:01.000Z",
    "article:author": "https://towardsdatascience.com/author/niklas_lang/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_000.txt",
      "chars": 73
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_001.txt",
      "chars": 40
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_002.txt",
      "chars": 30
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_003.txt",
      "chars": 147
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_004.txt",
      "chars": 70
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_005.txt",
      "chars": 43
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_006.txt",
      "chars": 80
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_007.txt",
      "chars": 51
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_008.txt",
      "chars": 96
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_009.txt",
      "chars": 106
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_010.txt",
      "chars": 43
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_011.txt",
      "chars": 43
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_012.txt",
      "chars": 37
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_013.txt",
      "chars": 35
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_014.txt",
      "chars": 109
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_015.txt",
      "chars": 23
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_016.txt",
      "chars": 34
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_017.txt",
      "chars": 61
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_018.txt",
      "chars": 60
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_019.txt",
      "chars": 65
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_020.txt",
      "chars": 61
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_021.txt",
      "chars": 22
    },
    {
      "language": "txt",
      "file": "attachments/8e5aa39323e8/code/block_022.txt",
      "chars": 122
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:52Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:52Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:52Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:52Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/data-engineering/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/niklas_lang/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/mastering-hadoop-part-1-installation-configuration-and-modern-big-data-strategies/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/big-data/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-processing/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/hadoop/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/exploring-the-contents-of-dlis-files-with-python-38585157dbac/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/andymcdonaldgeo/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/why-your-data-pipelines-need-closed-loop-feedback-control-76e28e3565f/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/jeff.b.chou/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/photos/grayscale-photography-of-man-in-striped-shirt-rgkKFn5Ysb0"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/data-engineer"
    },
    {
      "type": "external",
      "href": "https://databasecamp.de/en/data/apache-hdfs"
    },
    {
      "type": "external",
      "href": "https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html"
    },
    {
      "type": "external",
      "href": "https://databasecamp.de/en/data/apache-sparks"
    },
    {
      "type": "external",
      "href": "https://databasecamp.de/en/data/api-en"
    },
    {
      "type": "external",
      "href": "https://databasecamp.de/en/data/xml-file"
    },
    {
      "type": "external",
      "href": "https://databasecamp.de/en/data/kubernetes-basics"
    },
    {
      "type": "external",
      "href": "https://hadoop.apache.org/releases.html"
    },
    {
      "type": "external",
      "href": "https://hadoop.apache.org/docs/current/hadoop-project-dist/"
    },
    {
      "type": "external",
      "href": "https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Mastering+Hadoop%2C+Part+2%3A+Getting+Hands-On+%E2%80%94+Setting+Up+and+Scaling+Hadoop&u=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Mastering+Hadoop%2C+Part+2%3A+Getting+Hands-On+%E2%80%94+Setting+Up+and+Scaling+Hadoop&url=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Mastering+Hadoop%2C+Part+2%3A+Getting+Hands-On+%E2%80%94+Setting+Up+and+Scaling+Hadoop&url=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:31:51Z",
  "source_url": "https://towardsdatascience.com/mastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\n# Mastering Hadoop, Part 2: Getting Hands-On — Setting Up and Scaling Hadoop\nUnderstanding Hadoop’s core components before installation and scaling\n[Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\nMarch 13, 202520 min read\n![Person in striped shirt plugging audio cords in](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Scaling-Hadoop-3.png)\nImage from [Unsplash](https://unsplash.com/photos/grayscale-photography-of-man-in-striped-shirt-rgkKFn5Ysb0%20%0A)\nNow that we’ve explored [Hadoop’s role and relevance](https://towardsdatascience.com/mastering-hadoop-part-1-installation-configuration-and-modern-big-data-strategies/), it’s time to show you how it works under the hood and how you can start working with it. To start, we are breaking down Hadoop’s core components — HDFS for storage, MapReduce for processing, YARN for resource management, and more. Then, we’ll guide you through installing Hadoop (both locally and in the cloud) and introduce some essential commands to help you navigate and operate your first Hadoop environment.\n## Which components are part of the Hadoop architecture?\nHadoop's architecture is designed to be resilient and error-free, relying on several core components that work together. These components divide large datasets into smaller blocks, making them easier to process and distribute across a cluster of servers. This distributed approach enables efficient data processing—far more scalable than a centralized 'supercomputer.'\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Scaling-Hadoop-1.png)\nHadoop Components | Source: Author\nThe basic components of Hadoop are:\nLearn this step by step with the interactive [Data Engineer](https://roadmap.sh/data-engineer) roadmap.\n  * **Hadoop Common** comprises basic libraries and functionalities that are required by the other modules.\n  * The [**Hadoop Distributed File System (HDFS)**](https://databasecamp.de/en/data/apache-hdfs) ensures that data is stored on different servers and enables a particularly large bandwidth.\n  * **Hadoop YARN** takes care of resource distribution within the system and redistributes the load when individual computers reach their limits.\n  * **MapReduce** is a programming model designed to make the processing of large amounts of data particularly efficient.\n\n\nIn 2020, Hadoop Ozone, which is used as an alternative to HDFS, was added to this basic architecture. It comprises a distributed object storage system that was specially designed for big data workloads to better handle modern data requirements, especially in the cloud environment.\n### HDFS (Hadoop Distributed File System)\nLet’s dive into HDFS, the core storage system of Hadoop, designed specifically to meet the demands of big data processing. The basic principle is that files are not stored as a whole on a central server, but are divided into blocks of 128MB or 256MB in size and then distributed across different nodes in a computer cluster.\nTo ensure data integrity, each block is replicated three times across different servers. If one server fails, the system can still recover from the remaining copies. This replication makes it easy to fall back on another node in the event of a failure.\nAccording to its [documentation](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html), Hadoop pursues the following goals with the use of HDFS:\n  * **Fast recovery** from hardware failures by falling back on working components.\n  * Provision of **stream data processing**.\n  * **Big data framework** with the ability to process large data sets.\n  * **Standardized processes** with the ability to easily migrate to new hardware or software.\n\n\nApache Hadoop works according to the so-called master-slave principle. In this cluster, there is one node that takes on the role of the master. It distributes the blocks from the data set to various slave nodes and remembers which partitions it has stored on which computers. Only the references to the blocks, i.e. the metadata, are stored on the master node. If a master fails, there is a secondary name node that can take over.\nThe master within the Apache Hadoop Distributed File System is called a NameNode. The slave nodes, in turn, are the so-called DataNodes. The task of the DataNodes is to store the actual data blocks and regularly report the status to the NameNode that they are still alive. If a DataNode fails, the data blocks are replicated by other nodes to ensure sufficient fault tolerance.\nThe client saves files that are stored on the various DataNodes. In our example, these are located on racks 1 and 2. As a rule, there is only one DataNode per machine in a rack. Its primary task is to manage the data blocks in memory.\nThe NameNode, in turn, is responsible for remembering which data blocks are stored in which DataNode so that it can retrieve them on request. It also manages the files and can open, close, and, if necessary, rename them.\nFinally, the DataNodes carry out the actual read and write processes of the client. The client receives the required information from the DataNodes when a query is made. They also ensure the replication of data so that the system can be operated in a fault-tolerant manner.\n### MapReduce\nMapReduce is a programming model that supports the parallel processing of large amounts of data. It was originally developed by Google and can be divided into two phases:\n  * **Map** : In the map phase, a process is defined that can transform the input data into key-value pairs. Several mappers can then be set up to process a large amount of data simultaneously to enable faster processing.\n  * **Reduce** : The Reduce phase starts after all mappers have finished and aggregates all values that have the same key. The aggregation can involve various functions, such as the sum or the determination of the maximum value. Between the end of the Map phase and the start of the Reduce phase, the data is shuffled and sorted according to the keys.\n\n\nA classic application for the MapReduce mechanism is word counting in documents, such as the seven Harry Potter volumes in our example. The task is to count how often the words “Harry” and “Potter” occur. To do this, in the map phase, each word is split into a key-value pair with the word as the key and the number one as the value, as the word has occurred once.\nThe positive aspect of this is that this task can run in parallel and independently of each other, so that, for example, a mapper can run for each band or even for each page individually. This means that the task is parallelized and can be implemented much faster. The scaling depends only on the available computing resources and can be increased as required if the appropriate hardware is available. The output of the map phase could look like this, for example:\nmarkup\n\n```\n[(„Harry“, 1), („Potter“, 1), („Potter“, 1), („Harry“, 1), („Harry”, 1)]\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Scaling-Hadoop-2.png)\nMapReduce using the example of word counts in Harry Potter books | Source: Author\nOnce all mappers have finished their work, the reduce phase can begin. For the word count example, all key-value pairs with the keys “Harry” and “Potter” should be grouped and counted. \nThe grouping produces the following result:\nbash\n\n```\n[(„Harry“, [1,1,1]), („Potter“, [1,1])]\n```\n\nThe grouped result is then aggregated. As the words are to be counted in our example, the grouped values are added together:\nbash\n\n```\n[(„Harry“, 3), („Potter“, 2)]\n```\n\nThe advantage of this processing is that the task can be parallelized and at the same time only minimal file movement takes place. This means that even large volumes can be processed efficiently.\nAlthough many systems continue to use the MapReduce program, as used in the original Hadoop structure, more efficient frameworks, such as [Apache Spark](https://databasecamp.de/en/data/apache-sparks), have also been developed in the meantime. We will go into this in more detail later in the article.\n### YARN (Yet Another Resource Negotiator)\nYARN (Yet Another Resource Negotiator) manages the hardware resources within the cluster. It separates resource management from data processing, which allows multiple applications (such as MapReduce, Spark, and Flink) to run efficiently on the same cluster. It focuses on key functions such as:\n  * Management of performance and memory resources, such as CPU or SSD storage space.\n  * Distribution of free resources to running processes, for example, MapReduce, Spark, or Flink.\n  * Optimization and parallelization of job execution.\n\n\nSimilar to HDFS, YARN also follows a master-slave principle. The Resource Manager acts as the master and centrally monitors all resources in the entire cluster. It also allocates the available resources to the individual applications. The various node managers serve as slaves and are installed on each machine. They are responsible for the containers in which the applications run and monitor their resource consumption, such as memory space or CPU performance. These figures are fed back to the Resource Manager at regular intervals so that it can maintain an overview.\nAt a high level, a request to YARN looks like this: the client calls the Resource Manager and requests the execution of an application. This then searches for available resources in the cluster and, if possible, starts a new instance of the so-called Application Master, which initiates and monitors the execution of the application. This in turn requests the available resources from the node manager and starts the corresponding containers. The calculation can now run in parallel in the containers and is monitored by the Application Master. After successful processing, YARN releases the resources used for new jobs.\n### Hadoop common\nHadoop Common can be thought of as the foundation of the complete Hadoop ecosystem on which the main components can be built. It contains basic libraries, tools, and configuration files that can be used by all Hadoop components. The main components include:\n  * **Common libraries and utilities** : Hadoop Common provides a set of Java libraries, [APIs](https://databasecamp.de/en/data/api-en), and utilities needed to run the cluster. This includes, for example, mechanisms for communication between the nodes in the cluster or support for different serialization formats, such as Avro. Interfaces required for file management in HDFS or other file systems are also included.\n  * **Configuration management** : Hadoop is based on a large number of [XML](https://databasecamp.de/en/data/xml-file)-based configuration files, which define the main system parameters that are essential for operation. One central aspect is the network parameters required to control the machines in the cluster. In addition, the permitted storage locations for HDFs are defined here or the maximum resource sizes, such as the usable storage space, are determined.\n  * **Platform independence** : Hadoop was originally developed specifically for Linux environments. However, it can also be extended to other operating systems with the help of Hadoop Common. This includes native code support for additional environments, such as macOS or Windows.\n  * **Tools for I/O (input/output)** : A big data framework processes huge volumes of data that need to be stored and processed efficiently. The necessary building blocks for various file systems, such as TextFiles or Parquet, are therefore stored in Hadoop Common. It also contains the functionalities for the supported compression methods, which ensure that storage space is saved and processing time is optimized.\n\n\nThanks to this uniform and central code base, Hadoop Common provides improved modularity within the framework and ensures that all components can work together seamlessly.\n### Hadoop Ozone\nHadoop Ozone is a distributed object storage system that was introduced as an alternative to HDFS and was developed specifically for big data workloads. HDFS was originally designed for large files with many gigabytes or even terabytes. However, it quickly reaches its limits when a large number of small files need to be stored. The main problem is the limitation of the NameNode, which stores metadata in RAM and, therefore, encounters memory problems when billions of small files are kept.\nIn addition, HDFS is designed for classic Hadoop use within a computing cluster. However, current architectures often use a hybrid approach with storage solutions in the cloud. Hadoop Ozone solves these problems by providing a scalable and flexible storage architecture that is optimized for [Kubernetes](https://databasecamp.de/en/data/kubernetes-basics) and hybrid cloud environments.\nUnlike HDFS, where a NameNode handles all file metadata, Hadoop Ozone introduces a more flexible architecture that doesn't rely on a single centralized NameNode, improving scalability. Instead, it uses the following components: \n  * The **Ozone Manager** corresponds most closely to the HDFS NameNode, but only manages the bucket and volume metadata. It ensures efficient management of the objects and is also scalable, as not all file metadata has to be kept in RAM.\n  * The **Storage Container Manager (SCM)** can best be imagined as the DataNode in HDFS and it has the task of managing and replicating the data in so-called containers. Various replication strategies are supported, such as triple copying or erasure coding to save space.\n  * The **Ozone 3 Gateway** has an S3-compatible API so it can be used as a replacement for Amazon S3. This means that applications developed for AWS S3 can be easily connected to Ozone and interact with it without the need for code changes.\n\n\nThis structure gives Hadoop Ozone various advantages over HDFS, which we have briefly summarized in the following table:  \n| **Attribute**  | **Hadoop Ozone**  | **HDFS**  |  \n| --- | --- | --- |  \n| Storage Structure  | Object-based (buckets & keys)  | Block-based (files & blocks)  |  \n| Scalability  | Millions to billions of small files  | Problems with many small files  |  \n| NameNode - Dependency  | No central NameNode & scaling possible  | NameNode is bottleneck  |  \n| Cloud Integration  | Supports S3 API, Kubernetes, multi-cloud  | Strongly tied to the Hadoop Cluster  |  \n| Replication Strategy  | Classic 3-fold replication or erasure coding  | Only 3-fold replication  |  \n| Applications  | Big data, Kubernetes, hybrid cloud, S3 replacement  | Traditional Hadoop workloads  |  \nHadoop Ozone is a powerful extension of the ecosystem and enables the implementation of hybrid cloud architectures that would not have been possible with HDFS. It is also easy to scale as it is no longer dependent on a central name node. This means that big data applications with many, but small, files, such as those used for sensor measurements, can also be implemented without any problems.\n## How to start with Hadoop?\nHadoop is a robust and scalable big data framework that powers some of the world's largest data-driven applications. While it can seem overwhelming for beginners due to its many components, this guide will walk you through the first steps to get started with Hadoop in simple, easy-to-follow stages.\n**Installation of Hadoop**\nBefore we can start working with Hadoop, we must first install it in our respective environment. In this chapter, we differentiate between several scenarios, depending on whether the framework is installed locally or in the cloud. At the same time, it is generally advisable to work on systems that use Linux or macOS as the operating system, as additional adaptations are required for Windows. In addition, Java should already be available, at least Java 8 or 11, and internal communication via SSH should be possible.\n**Local Installation of Hadoop**\nTo try out Hadoop on a local computer and familiarize yourself with it, you can perform a single-node installation so that all the necessary components run on the same computer. Before starting the installation, you can check the latest version you want to install at <https://hadoop.apache.org/releases.html>, in our case this is version 3.4.1. If a different version is required, the following commands can simply be changed so that the version number in the code is adjusted.\nWe then open a new terminal and execute the following code, which downloads the specified version from the Internet, unpacks the directory, and then changes to the unpacked directory.\nbash\n\n```\n&#119;&#103;&#101;&#116; https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz\ntar -xvzf hadoop-3.4.1.tar.gz\ncd hadoop-3.4.1\n```\n\nIf there are errors in the first line, this is most likely due to a faulty link and the version mentioned may no longer be accessible. A more up-to-date version should be used and the code executed again. The installation directory has a size of about one gigabyte.\nThe environment variables can then be created and set, which tells the system under which directory Hadoop is stored on the computer. The `PATH` variable then allows Hadoop commands to be executed from anywhere in the terminal without having to set the full path for the Hadoop installation.\nbash\n\n```\nexport HADOOP_HOME=~/hadoop-3.4.1 \nexport PATH=$PATH:$HADOOP_HOME/bin\n```\n\nBefore we start the system, we can change the basic configuration of Hadoop, for example, to define specific directories for HDFS or specify the replication factor. There are a total of three important configuration files that we can adjust before starting:\n  * `core-site.xml` configures basic Hadoop settings, such as the connection information for multiple nodes.\n  * `hdfs-site.xml` contains special parameters for the HDFS setup, such as the typical directories for data storage or the replication factor, which determines how many replicas of the data are stored.\n  * `yarn-site.xml` configures the YARN component, which is responsible for resource management and job scheduling.\n\n\nFor our local test, we can adjust the HDFS configuration so that the replication factor is set to 1, as we are only working on one server, and replication of the data is, therefore, not useful. To do this, we use a text editor, in our case `nano`, and open the configuration file for HDFS:\nbash\n\n```\nnano $HADOOP_HOME/etc/hadoop/hdfs-site.xml\n```\n\nThe file then opens in the terminal and probably does not yet have any entries. A new XML with the property key can then be added within the configuration area:\nbash\n\n```\n<property> \n    <name>dfs.replication</name> \n    <value>1</value> \n</property>\n```\n\nVarious properties can then be set according to this format. The different keys that can be specified in the configuration files, including the permitted values, can be found at <https://hadoop.apache.org/docs/current/hadoop-project-dist/>. For HDFS, this overview can be seen [here](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml).\nNow that the configuration has been completed, Hadoop can be started. To do this, HDFS is initialized, which is the first important step after a new installation, and the directory that is to be used as the NameNode is formatted. The next two commands then start HDFS on all nodes that are configured in the cluster and the resource management YARN is started.\nbash\n\n```\nhdfs namenode -format \nstart-dfs.sh \nstart-yarn.sh\n```\n\nProblems may occur in this step if Java has not yet been installed. However, this can easily be done with the corresponding installation. In addition, when I tried this on macOS, the NameNode and DataNode of HDFS had to be started explicitly:\nbash\n\n```\n~/hadoop-3.4.1/bin/hdfs --daemon start namenode\n~/hadoop-3.4.1/bin/hdfs --daemon start datanode\n```\n\nFor YARN, the same procedure works for the Resource and NodeManager:\nbash\n\n```\n~/hadoop-3.4.1/bin/yarn --daemon start resourcemanager\n~/hadoop-3.4.1/bin/yarn --daemon start nodemanager\n```\n\nFinally, the running processes can be checked with the _jps_ command to see whether all components have been started correctly.\n## Hadoop installation in a distributed system\nFor resilient and productive processes, Hadoop is used in a distributed environment with multiple servers, known as nodes. This ensures greater scalability and availability. A distinction is typically made between the following cluster roles:\n  * **NameNode** : This role stores the metadata and manages the file system (HDFS).\n  * **DataNode** : This is where the actual data is stored and the calculations take place.\n  * **ResourceManager & NodeManagers**: These manage the cluster resources for YARN.\n\n\nThe same commands that were explained in more detail in the last section can then be used on the individual servers. However, communication must also be established between them so that they can coordinate with each other. In general, the following sequence can be followed during installation:\n  1. Set up several Linux-based servers to be used for the cluster.\n  2. Set up SSH access between the servers so that they can communicate with each other and send data.\n  3. Install Hadoop on each server and make the desired configurations.\n  4. Assign roles and define the NameNodes and DataNodes in the cluster.\n  5. Format NameNodes and then start the cluster.\n\n\nThe specific steps and the code to be executed then depend more on the actual implementation.\n## Hadoop installation in the cloud\nMany companies use Hadoop in the cloud to avoid having to operate their own cluster, potentially save costs, and also be able to use modern hardware. The various providers already have predefined programs with which Hadoop can be used in their environments. The most common Hadoop cloud services are:\n  * **AWS EMR (Elastic MapReduce)** : This program is based on Hadoop and, as the name suggests, also uses MapReduce, which allows users to write their programs in Java that process and store large amounts of data in a distributed manner. The cluster runs on virtual servers in the Amazon Elastic Compute Cloud (EC2) and stores the data in the Amazon Simple Storage Service (S3). The keyword “Elastic” comes from the fact that the system can change dynamically to adapt to the required computing power. Finally, AWS EMR also offers the option of using other Hadoop extensions such as Apache Spark or Apache Presto.\n  * **Google Dataproc** : Google's alternative is called Dataproc and enables a fully managed and scalable Hadoop cluster in the Google Cloud. It is based on BigQuery and uses Google Cloud Storage for data storage. Many companies, such as Vodafone and Twitter are already using this system.\n  * **Azure HDInsight** : The Microsoft Azure Cloud offers HDInsight for complete Hadoop use in the cloud and also provides support for a wide range of other open-source programs.\n\n\nThe overall advantage of using the cloud is that no manual installation and maintenance work is required. Several nodes are used automatically and more are added depending on the computing requirements. For the customer, the advantage of automatic scaling is that costs can be controlled and only what is used is paid for.\nWith an on-premise cluster, on the other hand, the hardware is usually set up in such a way that it is still functional even at peak loads so that the entire hardware is not required for a large part of the time. Finally, the advantage of using the cloud is that it makes it easier to integrate other systems that run with the same provider, for example.\n## Basic Hadoop commands for beginners\nRegardless of the architecture selected, the following commands can be used to perform very general and frequently recurring actions in Hadoop. This covers all areas that are required in an ETL process in Hadoop.\n  * **Upload File to HDFS** : To be able to execute an HDFS command, the beginning `hdfs dfs` is always required. You use put to define that you want to upload a file from the local directory to HDFS. The `local_file.txt` describes the file to be uploaded. To do this, the command is either executed in the directory of the file or the complete path to the file is added instead of the file name. Finally, use `/user/hadoop/` to define the directory in HDFS in which the file is to be stored.\n\n\nbash\n\n```\nhdfs dfs -put local_file.txt /user/hadoop/\n```\n\n  * **List files in HDFS** : You can use `-ls` to list all files and folders in the HDFS directory /user/hadoop/ and have them displayed as a list in the terminal.\n\n\nbash\n\n```\nhdfs dfs -put local_file.txt /user/hadoop/\n```\n\n  * **Download file from HDFS** : The `-get` parameter downloads the file `/user/hadoop/file.txt` from the HDFS directory to the local directory. The dot _._ indicates that the file is stored in the current local directory in which the command is being executed. If this is not desired, you can define a corresponding local directory instead.\n\n\nbash\n\n```\nhdfs dfs -get /user/hadoop/file.txt \n```\n\n  * **Delete files in HDFS** : Use _-rm_ to delete the file _/user/hadoop/file.txt_ from the HDFS directory. This command also automatically deletes all replications that are distributed across the cluster.\n\n\nbash\n\n```\nhdfs dfs -rm /user/hadoop/file.txt\n```\n\n  * **Start MapReduce command (process data)** : MapReduce is the distributed computing model in Hadoop that can be used to process large amounts of data. Using `hadoop jar` indicates that a Hadoop job with a “.jar” file is to be executed. The corresponding file containing various MapReduce programs is located in the directory `/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar`. From these examples, the `wordcount` job is to be executed, which counts the words occurring in a text file. The data to be analyzed is located in the HDFS directory `/input` and the results are then to be stored in the directory `output/`.\n\n\nbash\n\n```\nhadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount input/ output/\n```\n\n  * **Monitor the progress of a job** : Despite the distributed computing power, many MapReduce jobs take a certain amount of time to run, depending on the amount of data. Their status can therefore be monitored in the terminal. The resources and running applications can be displayed using YARN. To be able to execute a command in this system, we start with the command _yarn,_ and with the help of application-list we get a list of all active applications. Various information can be read from this list, such as the unique ID of the applications, the user who started them, and the progress in %.\n\n\nbash\n\n```\nyarn application -list\n```\n\n  * **Display logs of a running job** : To be able to delve deeper into a running process and identify potential problems at an early stage, we can read out the logs. The logs command is used for this, with which the logs of a specific application can be called up. The unique application ID is utilized to define this application. To do this, the APP_ID must be replaced by the actual ID in the following command, and the greater than and less than signs must be removed.\n\n\nbash\n\n```\nyarn logs -applicationId <APP_ID>\n```\n\nWith the help of these commands, data can already be saved in HDFS, and MapReduce jobs can also be created. These are the central actions for filling the cluster with data and processing it.\n## Debugging & logging in Hadoop\nFor the cluster to be sustainable in the long term and to be able to read out errors, it is important to master basic debugging and logging commands. As Hadoop is a distributed system, errors can occur in a wide variety of components and nodes. It is therefore essential that you are familiar with the corresponding commands to quickly find and switch off errors.\nDetailed log files for the various components are stored in the `$HADOOP_HOME/logs`directory. The log files for the various servers and components can then be found in their subdirectories. The most important ones are:\n  * **NameNode-Logs** contains information about the HDFS metadata and possible connection problems:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/hadoop-hadoop-namenode-<hostname>.log \n```\n\n  * **DataNode logs** show problems with the storage of data blocks:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/hadoop-hadoop-datanode-<hostname>.log\n```\n\n  * YARN **ResourceManager logs** reveal possible resource problems or errors in job scheduling:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/yarn-hadoop-resourcemanager-<hostname>.log\n```\n\n  * **NodeManager logs** help with debugging executed jobs and their logic:\n\n\nbash\n\n```\ncat $HADOOP_HOME/logs/yarn-hadoop-nodemanager-<hostname>.log\n```\n\nWith the help of these logs, specific problems in the processes can be identified and possible solutions can be derived from them. However, if there are problems in the entire cluster and you want to check the overall status across individual servers, it makes sense to carry out a detailed cluster analysis with the following command:\nbash\n\n```\nhdfs dfsadmin -report\n```\n\nThis includes the number of active and failed DataNodes, as well as the available and occupied storage capacities. The replication status of the HDFS files is also displayed here and additional runtime information about the cluster is provided. An example output could then look something like this:\nbash\n\n```\nConfigured Capacity: 10 TB\nDFS Used: 2 TB\nRemaining: 8 TB\nNumber of DataNodes: 5\nDataNodes Available: 4\nDataNodes Dead: 1\n```\n\nWith these first steps, we have learned how to set up a Hadoop in different environments, store and manage data in HDFS, execute MapReduce jobs, and read the logs to detect and fix errors. This will enable you to start your first project in Hadoop and gain experience with big data frameworks.\nIn this part, we covered the core components of Hadoop, including HDFS, YARN, and MapReduce. We also walked through the installation process, from setting up Hadoop in a local or distributed environment to configuring key files such as core-site.xml and hdfs-site.xml. Understanding these components is crucial for efficiently storing and processing large datasets across clusters.\nIf this basic setup is not enough for your use case and you want to learn how you can extend your Hadoop cluster to make it more adaptable and scalable, then our next part is just right for you. We will dive deeper into the large Hadoop ecosystem including tools like Apache Spark, HBase, Hive, and many more that can make your cluster more scalable and adaptable. Stay tuned!\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657097324_l9dcjv.jpeg)\nWritten By\n## [Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\n[See all from Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Big Data](https://towardsdatascience.com/tag/big-data/)[Data Processing](https://towardsdatascience.com/tag/data-processing/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Hadoop](https://towardsdatascience.com/tag/hadoop/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F&title=Mastering%20Hadoop%2C%20Part%202%3A%20Getting%20Hands-On%20%E2%80%94%20Setting%20Up%20and%20Scaling%20Hadoop)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F&title=Mastering%20Hadoop%2C%20Part%202%3A%20Getting%20Hands-On%20%E2%80%94%20Setting%20Up%20and%20Scaling%20Hadoop)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmastering-hadoop-part-2-getting-hands-on-setting-up-and-scaling-hadoop%2F&text=Mastering%20Hadoop%2C%20Part%202%3A%20Getting%20Hands-On%20%E2%80%94%20Setting%20Up%20and%20Scaling%20Hadoop)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/0ioK6L1cZz-MO_GES-scaled.jpg)](https://towardsdatascience.com/exploring-the-contents-of-dlis-files-with-python-38585157dbac/)\n## [Exploring the Contents of DLIS Files with Python](https://towardsdatascience.com/exploring-the-contents-of-dlis-files-with-python-38585157dbac/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWorking with Pandas and dlisio to Explore Well Log Data\n[Andy McDonald](https://towardsdatascience.com/author/andymcdonaldgeo/)\nJuly 14, 20238 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/0aQ5RPqHL5GbJRkpk.png)](https://towardsdatascience.com/why-your-data-pipelines-need-closed-loop-feedback-control-76e28e3565f/)\n## [Why Your Data Pipelines Need Closed-Loop Feedback Control](https://towardsdatascience.com/why-your-data-pipelines-need-closed-loop-feedback-control-76e28e3565f/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nRealities of company and cloud complexities require new levels of control and autonomy to meet business goals at scale\n[Jeff Chou](https://towardsdatascience.com/author/jeff.b.chou/)\nSeptember 10, 20239 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/03/0AN9iRrUBbRZAnvqa-scaled.jpg)](https://towardsdatascience.com/building-pipelines-in-apache-airflow-for-beginners-58f87a1512d5/)\n## [Building Pipelines In Apache Airflow - For Beginners](https://towardsdatascience.com/building-pipelines-in-apache-airflow-for-beginners-58f87a1512d5/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA quick and simple demo for running DAGs on Airflow\n[Aashish Nair](https://towardsdatascience.com/author/aashishnair/)\nMarch 15, 20234 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/0yMbTTTVP7IUe-doG.jpg)](https://towardsdatascience.com/extract-load-transform-learn-and-serve-getting-value-from-data-4f08d70bf343/)\n## [Extract & Load, Transform, Learn, and Serve - Getting Value from Data](https://towardsdatascience.com/extract-load-transform-learn-and-serve-getting-value-from-data-4f08d70bf343/)\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\nWe've all heard about ELT and ETL, but what about ELTLS?\n[Raghotham Murthy](https://towardsdatascience.com/author/raghotham/)\nJune 9, 20217 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/1rv6t0nVkivXFkqgimO4C6A-scaled.jpeg)](https://towardsdatascience.com/how-i-use-tkinter-python-library-to-run-algorithms-6be85f8f4e5b/)\n## [How I use Tkinter Python Library to Run Algorithms](https://towardsdatascience.com/how-i-use-tkinter-python-library-to-run-algorithms-6be85f8f4e5b/)\n[Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/)\nIf you're running several python scripts for your ETL or algorithms - Tkinter is a python library that can provide GUI elements and...\n[Kathleen Lara](https://towardsdatascience.com/author/itskathleenlara/)\nMarch 10, 20223 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/09/1XEMkPRGhR8D3KSaEwZsmlg.jpeg)](https://towardsdatascience.com/what-i-learned-building-platforms-at-stitch-fix-fc5e0ec72c86/)\n## [What I learned building platforms at Stitch Fix](https://towardsdatascience.com/what-i-learned-building-platforms-at-stitch-fix-fc5e0ec72c86/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nFive lessons\n[Stefan Krawczyk](https://towardsdatascience.com/author/stefan.krawczyk/)\nSeptember 26, 202223 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1iXE5khfhaitDWnVvwKCKTg-scaled.jpeg)](https://towardsdatascience.com/you-do-need-math-for-machine-learning-cf934a607960/)\n## [You DO need math for Machine Learning](https://towardsdatascience.com/you-do-need-math-for-machine-learning-cf934a607960/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nDon't skip the most unappreciated skill in ML. Write code AND learn the theory.\n[Sarem Seitz](https://towardsdatascience.com/author/sarem-seitz/)\nJuly 2, 202112 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/15UIxC5-70TnQNcK1a6PMWw-scaled.jpeg)](https://towardsdatascience.com/dbt-cli-model-selection-52ddd038d8b2/)\n## [Model Selection For dbt CLI](https://towardsdatascience.com/dbt-cli-model-selection-52ddd038d8b2/)\n[Programming](https://towardsdatascience.com/category/programming/)\nA complete cheatsheet for selecting specific models when running dbt commands\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nJanuary 25, 20238 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/14V7uvW0Rcnap8BiNAJvtcA.jpeg)](https://towardsdatascience.com/how-to-go-from-a-python-newbie-to-a-google-certified-tensorflow-developer-under-two-months-3d1535432284/)\n## [How to go from a Python newbie to a Google Certified TensorFlow Developer under two months.](https://towardsdatascience.com/how-to-go-from-a-python-newbie-to-a-google-certified-tensorflow-developer-under-two-months-3d1535432284/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA learning journey (with tips and tricks) on how I got certified as a TensorFlow Developer by Google within two months.\n[Grady Matthias Oktavian](https://towardsdatascience.com/author/radvian/)\nSeptember 1, 202012 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "body_base_path": "attachments/8e5aa39323e8/pages/",
  "body_markdown_path": "attachments/8e5aa39323e8/pages/archived-body.md",
  "local_html": "attachments/8e5aa39323e8/pages/archived-body.html",
  "body_source": "prior_capture"
}