Enterprise open source Toolkit

Yahoo and uni team up on distributed computing

Suzanne Tindal ZDNet Australia

Published: 13 Nov 2007 10:10 GMT

Yahoo has announced a programme to encourage the development of software around distributed computing.

The open-source programme will see Yahoo push work around Hadoop — an environment that enables its users to process massive amounts of data via distributed computing, which divides an application into many small fragments of work to be executed on large numbers of separate computers linked in a network.

The US's Carnegie Mellon University is now involved in the programme and will run Hadoop on Yahoo's supercomputing cluster, the M45, which has a peak performance of more than 27 teraflops.

Systems software researchers from the university will study the system's performance, while computer science professors will use the cluster to tackle information-retrieval and graph problems. The cluster will also be used to work on large-scale computer graphics, natural language processing, and machine translation problems.

"We are excited about collaborating with Yahoo on systems software research... and jointly contributing back to the open-source community," Randall E Bryant, dean of the School of Computer Science at Carnegie Mellon, said in a statement.

Homebrew blog

Tell us about your home tech and you could win a car

Blog about your pet project for a chance to win a Toyota Prius

Read this

Yahoo is not the only company to work in the area of handling of large data sets.

Google and IBM have provided hardware, software and services to academics in multiple universities to improve computer science students' knowledge of highly parallel computing practices used to work with large data sets, while Australia's Commonwealth Scientific and Industrial Research Organisation (CSIRO) announced the start of a terabyte science project last week to research methods of dealing with large amounts of data.