About

The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing.

"Hadoop" typically refers to the software in the project that implements the map-reduce data analysis framework, plus the distributed file system (HDFS) that underlies it.

Since version 0.23, Hadoop disposes of a standalone resource manager : yarn.

This resource manager makes it easier to use other modules alongside with the MapReduce engine, such as :

• Ambari, A web-based tool for provisioning, managing, and
  monitoring Apache Hadoop clusters which includes support for Hadoop
  HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster
  health such as heatmaps and ability to view MapReduce, Pig and Hive
  applications visually along with features to diagnose their
  performance characteristics in a user-friendly manner.
• Avro, a data serialization system based on JSON schemas.
• Cassandra, a replicated, fault-tolerant, decentralized and scalable database system.
• Chukwa: A data collection system for managing large distributed systems.
• HBase, A scalable, distributed database that supports structured data storage for large tables.
• Hive, A data warehouse infrastructure that provides data summarization and ad hoc querying.
• Mahout, a library of machine learning algorithms compatible with M/R paradigm.
• Pig, a platform/programming language for authoring parallelizable jobs
• Storm, a system for real-time and stream processing
• ZooKeeper, a system for coordinating distributed nodes, similar to Google's Chubby
• Oozie, a workflow scheduler system to manage Apache Hadoop jobs.
• Spark, a fast and general engine for large-scale data processing.
• Flink, a fast and reliable large-scale data processing engine.

Recommended reference sources:

• Hive Language Reference

Commercial support is available from a variety of companies.