http://www.bibsonomy.org/burst/user/neilernst/extractionBibSonomy BuRST Feed for /user/neilernst/extraction2008-10-16T06:50:58+02:00An overview of "Information extraction"http://www.bibsonomy.org/bibtex/2ee6de663fac26a3777328c769ca3cc70/neilernstneilernst2008-02-18T18:11:47+01:00information unstructured extraction <span style="color:#555555;">A. <a href="http://www.bibsonomy.org/author/McCallum">McCallum</a> </span><em>Queue</em><em>3(9):48--57</em>(<em>2005</em>)Mon Feb 18 18:11:47 CET 2008Queue948--57Information extraction: distilling structured data from unstructured text32005information unstructured extraction In 2001 the U.S. Department of Labor was tasked with building a Web site that would help people find continuing education opportunities at community colleges, universities, and organizations across the country. The department wanted its Web site to support fielded Boolean searches over locations, dates, times, prerequisites, instructors, topic areas, and course descriptions. Ultimately it was also interested in mining its new database for patterns and educational trends. This was a major data-integration project, aiming to automatically gather detailed, structured information from tens of thousands of individual institutions every three months.The first and biggest problem was that much of the data wasn't available even in semi-structured form, much less normalized, structured form. Although some of the larger organizations had internal databases of their course listings, almost none of them had publicly available interfaces to their databases. The only universally available public interfaces were Web pages designed for human browsing. Unfortunately, but as expected, each organization used different text formatting. Some of these Web pages contained two-dimensional text tables; many others used a stylized collection of paragraphs for each course offering; still others had a single paragraph of English prose containing all the information about each course.The task thus required extracting structured information from English that had been formatted in a mixture of two-dimensional layout and free-running prose--a daunting technical challenge, but one that was ultimately solved successfully. More details about the solution follow, but first, let's place this problem in context.