IRS for Computer Character Sequences Filtration: a new software tool and algorithm to support the IRS at tokenization process
Q. Ahmad Al Badawi. International Journal of Advanced Computer Science and Applications(IJACSA), (2013)
Abstract
Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. A token is an instance of token a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. New software tool and algorithm to support the IRS at tokenization process are presented. Our proposed tool will filter out the three computer character Sequences: IP-Addresses, Web URLs, Date, and Email Addresses. Our tool will use the pattern matching algorithms and filtration methods. After this process, the IRS can start a new tokenization process on the new retrieved text which will be free of these sequences.
%0 Journal Article
%1 IJACSA.2013.040212
%A Ahmad Al Badawi, Qasem Abu Al-Haija
%D 2013
%J International Journal of Advanced Computer Science and Applications(IJACSA)
%K Filtration. Information Retrieval; Sequences Tokenization; and matching; pattern
%N 2
%T IRS for Computer Character Sequences Filtration: a new software tool and algorithm to support the IRS at tokenization process
%U http://ijacsa.thesai.org/
%V 4
%X Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. A token is an instance of token a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. New software tool and algorithm to support the IRS at tokenization process are presented. Our proposed tool will filter out the three computer character Sequences: IP-Addresses, Web URLs, Date, and Email Addresses. Our tool will use the pattern matching algorithms and filtration methods. After this process, the IRS can start a new tokenization process on the new retrieved text which will be free of these sequences.
@article{IJACSA.2013.040212,
abstract = {Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. A token is an instance of token a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. New software tool and algorithm to support the IRS at tokenization process are presented. Our proposed tool will filter out the three computer character Sequences: IP-Addresses, Web URLs, Date, and Email Addresses. Our tool will use the pattern matching algorithms and filtration methods. After this process, the IRS can start a new tokenization process on the new retrieved text which will be free of these sequences.},
added-at = {2014-02-21T08:00:08.000+0100},
author = {{Ahmad Al Badawi}, Qasem Abu Al-Haija},
biburl = {https://www.bibsonomy.org/bibtex/2211e7c88eb302f018a0f7f7c0d4483c5/thesaiorg},
interhash = {97d6a9904a8f3f1ada864e41e4eda169},
intrahash = {211e7c88eb302f018a0f7f7c0d4483c5},
journal = {International Journal of Advanced Computer Science and Applications(IJACSA)},
keywords = {Filtration. Information Retrieval; Sequences Tokenization; and matching; pattern},
number = 2,
timestamp = {2014-02-21T08:00:08.000+0100},
title = {{IRS for Computer Character Sequences Filtration: a new software tool and algorithm to support the IRS at tokenization process}},
url = {http://ijacsa.thesai.org/},
volume = 4,
year = 2013
}