Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets. At the present time, Pig's infrastructure layer consists of a compiler that produces sequences of Map-Reduce programs, for which large-scale parallel implementations already exist Pig's language layer currently consists of a textual language called Pig Latin, which has the following key properties: * Ease of programming. It is trivial to achieve parallel execution of simple, "embarrassingly parallel" data analysis tasks. * Optimization opportunities. The way in which tasks are encoded permits the system to optimize their execution automatically * Extensibility.
The lucky kids of JavaSchools are never going to get weird segfaults trying to implement pointer-based hash tables. They're never going to go stark, raving mad trying to pack things into bits. They'll never have to get their head around how, in a purely functional program, the value of a variable never changes.
Map-Reduce is on its way out. But we shouldn’t measure its importance in the number of bytes it crunches, but the fundamental shift in data processing architectures it helped popularise.
Sqoop is a tool designed to import data from relational databases into Hadoop. Sqoop uses JDBC to connect to a database. It examines each table’s schema and automatically generates the necessary classes to import data into the Hadoop Distributed File System (HDFS). Sqoop then creates and launches a MapReduce job to read tables from the database via DBInputFormat, the JDBC-based InputFormat. Tables are read into a set of files in HDFS. Sqoop supports both SequenceFile and text-based target and includes performance enhancements for loading data from MySQL.
P. Pantel, E. Crestan, A. Borkovsky, A. Popescu, and V. Vyas. EMNLP '09: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, page 938--947. Morristown, NJ, USA, Association for Computational Linguistics, (2009)
P. Ravindra, V. Deshpande, and K. Anyanwu. MDAC '10: Proceedings of the 2010 Workshop on Massive Data Analytics on the Cloud, page 1--6. New York, NY, USA, ACM, (2010)
C. Bellettini, M. Camilli, L. Capra, and M. Monga. Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), 2012 14th International Symposium on, page 295-302. IEEE Computer Society, (September 2012)
M. Becker, H. Mewes, A. Hotho, D. Dimitrov, F. Lemmerich, and M. Strohmaier. International Conference Companion on World Wide Web, page 17--18. Republic and Canton of Geneva, Switzerland, International World Wide Web Conferences Steering Committee, (2016)
M. Bayir, I. Toroslu, A. Cosar, and G. Fidan. WWW '09: Proceedings of the 18th international conference on World wide web, page 161--170. New York, NY, USA, ACM, (2009)
J. Urbani, S. Kotoulas, E. Oren, and F. van Harmelen. International Semantic Web Conference, volume 5823 of Lecture Notes in Computer Science, page 634-649. Springer, (2009)
A. Ghoting, P. Kambadur, E. Pednault, and R. Kannan. Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA, August 21-24, 2011, page 334-342. (2011)
F. Chierichetti, R. Kumar, and A. Tomkins. WWW '10: Proceedings of the 19th international conference on World wide web, page 231--240. New York, NY, USA, ACM, (2010)
F. Chierichetti, R. Kumar, and A. Tomkins. WWW '10: Proceedings of the 19th international conference on World wide web, page 231--240. New York, NY, USA, ACM, (2010)
C. Bellettini, M. Camilli, L. Capra, and M. Monga. Reachability Problems, volume 8169 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, (2013)