 |
VOOZH | about |
|
VOOZH | about |
This article was published as a part of the Data Science Blogathon
Regex is a shorthand for Regular Expression. It is a representation for a set, a set of strings. Say we have a list of emails and we want to check if they are in the correct format or not. One way is to check each and every mail manually but that’s not possible if the number of mails is quite high. So, regex here comes to your rescue. Whenever you want to check emails are valid or not, you can simply match them against the pattern. You’ll get true or false results whether the mail is in the correct format or not according to the regex pattern used.
What are Regular Expressions use for?
Some popular and very common use cases are:
These are very normal tasks when working with text data or solving a Natural Language Processing (NLP) problem. So, we will look at some popular uses of regex in NLP problems. You can call it your cheat sheet for NLP tasks. But before moving forward let’s have a look at some major Regular Expression functions.
Out of all these, we will be using re.findall to detect patterns.
So, let’s get started!
👁 Regex Cheatsheet For NLP tasks
Source: Google Images
To find a URL in a sentence using the following regex code:
def find_url(string):
text = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*(),]|
(?:%[0-9a-fA-F][0-9a-fA-F]))+',string)
#convert return value from list to string
return "".join(text)
example="I love spending time at https://www.leetcode.com/" find_url(example)
Output:
'https://www.leetcode.com/'
To find emoticons in a sentence following code can be used:
def findEmoji(text): emo_text=emoji.demojize(text) line=re.findall(r':(.*?):',emo_text) return line example="I love ⚽ very much 😁" findEmoji(example)
Output:
['soccer_ball', 'beaming_face_with_smiling_eyes']
Get email id from the sentence by using this code:
def findEmail(text): line = re.findall(r'[w.-]+@[w.-]+',str(text)) return ",".join(line)
example="Gaurav's gmail is [email protected]" findEmail(example)
Output:
'[email protected]'
You must be aware of Hash(#), it is used in Twitter to denote trends. So, to extract the hash words like #fifa or #cricket, use this code:
def findHash(text): line=re.findall(r'(?<=#)w+',text) return " ".join(line)
example="#Sushant is trending now in the world" findHash(example)
Output:
'Sushant'
@ – This symbol is used to mention someone in tweets. So, to extract words associated with @ symbol use this code:
def findAt(text): line=re.findall(r'(?<=@)w+',text) return " ".join(line)
example="@Ajit, please help me" findAt(example)
Output:
'Ajit'
Following regex, code is used to pick only numbers from a sentence.
def findNumber(text): line=re.findall(r'[0-9]+',text) return " ".join(line)
example="8853147 sq. km of area washed away in floods" findNumber(example)
Output:
'8853147'
Indian Mobile numbers have ten digits. So, the below regex is used to find an Indian phone number. You can modify the regex as per your need.
def findPhoneNumber(text):
line=re.findall(r"bd{10}b",text)
return "".join(line)
findPhoneNumber("9990001796 is a phone number of PMO office")
Output:
'9990001796'
Non-Alphanumeric characters are characters other than alphanumeric. Example: space,
percent sign, underscore, pipe, colon, semicolon, etc are non-alphanumeric characters. So, to find these in-text use this regex code.
def findNonalp(text):
line = re.findall("[^A-Za-z0-9 ]",text)
return line
example="Twitter has lots of @ and # in posts.(2021 year is not good)" findNonalp(example)
Output:
['@', '#', '.', '(', ')']
Following regex code can be used to extract years from 1940 till 2020.
def findYear(text): line=re.findall(r"b(19[40][0-9]|20[0-1][0-9]|2020)b",text) return line
example="My DOB year is 1998." findYear(example)
Output:
['1998']
To find these marks in your text use the below code:
def find_punct(text):
line = re.findall(r'[!"$%&'()*+,-./:;=#@?[\]^_`{|}~]*', text)
string="".join(line)
return list(string)
example="Corona virus kiled #24506 people. #Corona is un(tolerable)" print(find_punct(example))
Output:
['#', '.', '#', '(', ')']
Removes repetitive characters from words.
def rep(text): grp = text.group(0) if len(grp) > 1: return grp[0:1] # change the value here on repetition def unique_char(rep,sentence): convert = re.sub(r'(w)1+', rep, sentence) return convert
example="heyyy this is a verrrry loong texttt" unique_char(rep,example)
Output:
'hey this is a very long text'
Find numbers that are greater than 930. You can modify it as per your use.
def num_great(text):
line=re.findall(r'9[3-9][0-9]|[1-9]d{3,}',text)
return " ".join(line)
example="Height of this bridge is 935m. Width of this bridge is 30 metre. It used 9274kg of steel." num_great(example)
Output:
'935 9274'
Find numbers less than 930 in the input text.
def num_less(text): only_num=[] for i in text.split(): line=re.findall(r'^(9[0-2][0-0]|[1-8][0-9][0-9]|[1-9][0-9]|[0-9])$',i) only_num.append(line) all_num=[",".join(x) for x in only_num if x != []] return " ".join(all_num)
example="There are some countries where less than 920 cases exist with 1100 observations" num_less(example)
Output:
'920'
Find the date in this format from the input text.
def findDates(text):
line=re.findall(r'b(1[0-2]|0[1-9])/(3[01]|[12][0-9]|0[1-9])/([0-9]{4})b',text)
return line
example="Todays date is 06/21/2021 for format mm/dd/yyyy, not 31/09/2020" findDates(example)
Output:
[('06', '21', '2021')]
The below code only extracts the words, alphabets, and nothing else from a given piece of text.
def onlyWords(text): line=re.findall(r'b[^dW]+b', text) return " ".join(line)
example="Harish reduced his weight from 100 Kg to 75 kg." onlyWords(example)
Output:
'Harish reduced his weight from Kg to kg.'
Extracts all the numbers and no other text.
def only_numbers(text): line=re.findall(r'bd+b', text) return " ".join(line)
example="Harish reduced his weight from 100 Kg to 75 kg." only_numbers(example)
Output:
'100 75'
If you want all sentences with a particular keyword. Use the below code:
def pick_only_key_sentence(text,keyword): line=re.findall(r'([^.]*'+keyword+'[^.]*)', text) return line
example="People are fighting with covid these days. Economy has fallen down. How will we survive covid" pick_only_key_sentence(example,'covid')
Output:
['People are fighting with covid these days', 'How will we survive covid']
Extract words starting with a capital letter from the input text.
def find_capital(text): line=re.findall(r'b[A-Z]w+', text) return line
example="Ajit Doval is the best National Security Advisor so far." find_capital(example)
Output:
['Ajit', 'Doval', 'National','Security','Advisor']
Most web scrapped data contain HTML tags. To remove that use below regex script:
def remove_tag(string):
text=re.sub('<.*?>','',string)
return text
example="Markdown sentences use <br> for breaks and <i> </i> for italics" remove_tag(example)
Output:
'Markdown sentences use for breaks and for italics'
Extract IP address from the text.
def ip_add(string):
text=re.findall('d{1,3}.d{1,3}.d{1,3}.d{1,3}',string)
return text
example="My public IP address is 165.19.120.1" ip_add(example)
Output:
['165.19.120.1']
Extract Mac address from the text.
def mac_add(string):
text=re.findall('(?:[0-9a-fA-F]:?){12}',string)
return text
example="MAC ADDRESSES of this TOSHIBA laptop is 00:21:27:b1:aa:xx." mac_add(example)
Output:
['00:21:27:b1:aa:xx ']
To Validate PAN Number use the below code.
Format: First 5 letters in CAPS+4 digits+Last letter in CAPS
def validPan(string):
text=re.findall(r'^([A-Z]){5}([0-9]){4}([A-Z]){1}$',string)
if text!=[]:
print("{} is valid PAN number".format(string))
else:
print("{} is not a valid PAN number".format(string))
validPan("ABCED3193P")
validPan("lEcGD012eg")
Output:
ABCED3193P is valid PAN number lEcGD012eg is not a valid PAN number
Find the percentage numbers from the text. For example: if the sentence is “I scored 85% in 12th class.”. In that case, output will be “85”.
def find_percent(string): text = re.findall(r'b(100|[1-9][0-9]|[0-9])%',string) return text
example="COVID recovery rate is now 76%. But death rate is 4%" find_percent(example)
Output:
['76', '4']
This code will return you the filename with its file format like “abc.png”, “math.pdf”, etc.
def find_files(string):
text=re.findall(r'([a-zA-Z0-9_]+).(jpg|png|gif|jpeg|pdf|ipynb|py)',string)
# add any required file extensions
all_files=[]
for i in range(len(text)):
all_files.append('.'.join(text[i]))
return all_files
example="This image file name is cheatsheet.png . Titanic.py file is most common among beginners." find_files(example)
Output:
['cheatsheet.png', 'Titanic.py']
So, these are some regex codes that can be used in NLP tasks directly. I hope you find this article helpful. Let’s connect on Linkedin.
Thanks for reading if you reached here :).
Happy coding!
The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
Love Programming, Blog writing and Poetry
GPT-4 vs. Llama 3.1 – Which Model is Better?
Llama-3.1-Storm-8B: The 8B LLM Powerhouse Surpa...
A Comprehensive Guide to Building Agentic RAG S...
Top 10 Machine Learning Algorithms in 2026
45 Questions to Test a Data Scientist on Basics...
90+ Python Interview Questions and Answers (202...
8 Easy Ways to Access ChatGPT for Free
Prompt Engineering: Definition, Examples, Tips ...
What is LangChain?
What is Retrieval-Augmented Generation (RAG)?
Edit
Resend OTP
Resend OTP in 45s
