What is the Efficient Way of Reading a Huge Text File?

In C++, reading a large text file efficiently requires a careful approach to ensure optimal performance in terms of memory usage and processing speed. In this article, we will learn how to read a huge text file efficiently in C++.

Read a Large Text File Efficiently in C++

The most efficient way to read a large text file is to read the file in chunks rather than line by line or one character at a time by using the combination of and to parse each chunk. This method significantly reduces the I/O operations, thereby improving the overall performance.

Approach

• Create an object for the text file using std::ifstream.
• Open the file using the file stream object by passing the path of the file to the file stream object.
• Read the file in chunks of specified size referred to as BUFFER_SIZE, and process each chunk.
• Use std::istringstreamto parse each chunk into lines.
• Print the non-empty lines returned by istringstream.
• Process any remaining data in the last chunk to ensure no data is left unprocessed.
• Close the file after all data in the file has been processed.

C++ Program to Read a Huge Text File

The below program illustrates how we can read a huge text file effectively in C++.

Output

Hello World
GeeksforGeeks
C++ Java Python
GoLang Rust JavaScript

Time Complexity: O(N), where N is the total number of characters in the text file.
Auxiliary Space: O(M), where M is the length of the longest line in the text file.