VOOZH about

URL: https://www.geeksforgeeks.org/python/python-nltk-tokenize-wordpuncttokenizer/

⇱ Python NLTK | tokenize.WordPunctTokenizer() - GeeksforGeeks


  • Courses
  • Tutorials
  • Interview Prep

Python NLTK | tokenize.WordPunctTokenizer()

Last Updated : 30 Sep, 2019
With the help of nltk.tokenize.WordPunctTokenizer()() method, we are able to extract the tokens from string of words or sentences in the form of Alphabetic and Non-Alphabetic character by using tokenize.WordPunctTokenizer()() method.
Syntax : tokenize.WordPunctTokenizer()() Return : Return the tokens from a string of alphabetic or non-alphabetic character.
Example #1 : In this example we can see that by using tokenize.WordPunctTokenizer()() method, we are able to extract the tokens from stream of alphabetic or non-alphabetic character. Output :
['GeeksforGeeks', '...$$&*', 'is', 'for', 'geeks']
Example #2 :
Output :
['The', 'price', 'of', 'burger', 'in', 'BurgerKing', 'is', 'Rs', '.', '36', '.']
Comment
Article Tags:
Article Tags: