How to Read a File Line by Line in Bash [5 Easy Methods]

Written by

Reviewed by

Last updated: July 24, 2024

Expert verified

SVG Image

TL;DR

To read a file line by line in Bash, you can follow these steps:

  1. Open the terminal.
  2. Create a while loop structure using while read line; do ; done.
  3. Use the < symbol followed by the file name to specify the file, like while read line; do echo $line; done < file.txt.
  4. Run the script to read the file line by line until it reaches the end.

To learn more about how to read files line by line in Bash, read this detailed guide now.

Reading files line by line in bash scripting is something you probably face often. Whether you’re dealing with large files, filtering data, or handling real-time inputs, it’s crucial to do it efficiently. In this post, I’ll show you why reading files line by line matters and guide you through different methods to get it done. You’ll find step-by-step instructions, tips on handling common errors, and best practices to ensure smooth file processing. By the end, you’ll have the skills to make your scripts more efficient and effective.

Why You Need to Read Files Line by Line in Bash Scripting

Reading files line by line in bash scripting is important for several reasons. Here are some key points:

1. Processing Large Files: When dealing with large files, reading the entire file at once can use a lot of memory and slow down your script. Reading line by line is more efficient because it uses less memory and allows your script to handle large files smoothly.

2. Line-by-Line Processing: Sometimes, you need to process each line of a file separately. For example, you might need to:

  • Extract specific information from each line.
  • Perform different actions based on the content of each line.
  • Collect data from each line for further processing.

3. Filtering Data: Reading files line by line makes it easier to filter data. You can check each line for certain keywords or patterns and only process the lines that match your criteria. This is useful for tasks like:

  • Searching for errors in log files.
  • Extracting specific records from data files.
  • Ignoring lines that don’t contain relevant information.

4. Real-Time Data Processing: In some cases, you might need to process data as it is being written to a file. Reading the file line by line allows your script to handle new data as it arrives, which is essential for tasks like monitoring logs or processing data streams.

5. Flexibility and Control: Reading files line by line gives you more control over how you handle the data. You can:

  • Pause or stop reading based on certain conditions.
  • Skip lines that don’t meet specific criteria.
  • Process lines in a custom way that fits your needs.

How to Read a File Line by Line in Bash

To read a file line by line in bash, you can use a while loop with the read command. First, create a sample file using echo "your text" > filename.txt. Then, write a script: #!/bin/bash followed by while IFS= read -r line; do echo "$line"; done < filename.txt. Make the script executable with chmod +x scriptname.sh and run it using ./scriptname.sh. This loop reads each line of the file one at a time, ensuring efficient and clear processing.

Continue reading for detailed steps for this method and four other methods to read a file line by line in bash:

1. Use a While Loop

The while loop is a simple and effective way to read a file line by line in Bash. Here’s how to use this method:

  1. Launch the Terminal window and create a while loop structure using the “while” keyword.
while read line; do
<code to be executed>
done
  1. This sets up a loop that will continue as long as more input is read.
creating a loop structure read a file line by line in Bash
  1. You can include any code you want to execute within the loop for each file line. In this case, the code is $line, which will execute the contents of the variable line.
while read line; do
$line
done
  1. Your code in the Terminal should look like this:
applying condition for printing output
  1. To specify the file you want to read a file line by line in Bash, use the < symbol followed by the file name. In this example, the file is named “file.txt“.
while read line; do
echo $line
done < file.txt
  1. When the script runs, the while loop will read a file line by line in Bash until it reaches the end of the file.
fetching data from file
  1. The read command within the while loop can also be used to read input from other sources, such as user input. Instead of reading from a file, you can replace file.txt with another input source, like a variable or a command that produces output.
while read line; do
echo $line
done < $input_variable
  1. Here, $input_variable represents the variable containing the input or command that produces output.
getting data from other source using variable

2. Use a For Loop

The for loop is another efficient way to read a file line by line in Bash. Using the cat command within the for loop, you can iterate over each line of the file, executing code or performing operations on each line individually. Here’s how to use it: 

  1. In the Linux command prompt, navigate to the directory using the cd command containing the file you want to read. Then, use the ls command to list the files in the directory.
navigating to the directory
  1. Next, use the cat command with the for loop to read a file line by line in Bash:
for line in $(cat file.txt); do echo $line done
  1. Once this command is executed, you’ll see the following output:
using for loop
  1. You can also use the cat command within the for loop to concatenate multiple files into a single stream of data. To do this, simply replace “file.txt” with the names of the files you want to concatenate, separated by spaces. For instance, if you want to read two files, file1.txt and file2.txt, use the following command:
for line in $(cat file1.txt file2.txt); do echo $line; done
  1. This will concatenate the contents of file1.txt and file2.txt and read them line by line using the for loop.
reading line by line from multiple files

3. Use the Sed Command

The sed command is a powerful text editor that can be used to manipulate text in a file. Here’s how to use it to read a file line by line in Bash:

  1. In the Terminal, use the cd command to navigate to the directory containing the file you want to read. Then, execute the following command to read a file line by line in Bash:
sed -n 'p' file.txt
  1. This command tells sed to print the contents of each line in the file to the Terminal interface.
printing line by line using sed command
  1. You can also use the sed command in combination with other Bash commands to perform complex operations. For example, to find and replace text in a file, execute the following code:
sed 's/old_text/new_text/g' file.txt
  1. This command will search for all occurrences of old_text in the file and replace them with new_text.
replacing old text by new text using sed

4. Use the Awk Command

The awk command is a versatile and powerful tool for text processing in Bash. Here’s how to use it to read a file line by line in Bash:

  1. Head to the Terminal and navigate to the directory containing the file you want to read. Then, execute the following command to read a file line by line in Bash:
awk '{ print }' file.txt
  1. Now, the awk command will print the contents of each line in the file to the Terminal, and you’ll get the following output:
using awk command to print lines as it is
  1. Moreover, you can use the awk command to perform various text-processing tasks, such as filtering and manipulating text. For example, to print only the first field of each line in a file, run the following command:
awk '{ print $1 }' file.txt
  1. This command will print only the first field (i.e., the first word) of each line in the file.
print first field of line using awk

5. Using the read Command

Reading a file line by line is a fundamental task in many bash scripts. It allows for processing each line individually, which is useful for various automation and data processing tasks. In this guide, we’ll show you how to read a file line by line using the read command in bash.

  1. First, create a sample file named sample.txt with some content to read:
cat > sample.txt <<EOL

Line 1: Bash scripting is powerful.

Line 2: It allows for automation of tasks.

Line 3: Reading files line by line is a common task.

EOL
creating a text file and adding content in it
  1. Create a new script file named readfile.sh:
nano readfile.sh
creating a new script file and opening in nano editor
  1. Add the following script to readfile.sh:
#!/bin/bash

# Open the file using a file descriptor

while IFS= read -r line

do

  # Process each line

  echo "$line"

done < "sample.txt"

Explanation:

  • #!/bin/bash: Specifies the script should be run in the bash shell.
  • while IFS= read -r line: Reads each line from the file, preserving whitespace and special characters.
  • do … done < “sample.txt”: Redirects the contents of sample.txt into the while loop.
adding script code in script file in nano editor
  1. Save and close the file by pressing CTRL+X, then Y, and Enter.
saving and exiting the file
  1. Make your script executable:
chmod +x readfile.sh
making the script file executable
  1. Run the script to read the file line by line:
./readfile.sh

You should see the contents of sample.txt printed line by line in your terminal.

executing the script file

3 Common Issues When Reading Files Line by Line in Bash

Reading files line by line in Bash can sometimes be tricky. But don’t worry, understanding how to fix these problems can make working with any file a breeze. Here are the top three things that can go wrong and some easy steps you can follow to sort them out:

1. Improper File Permissions

If the file permissions are not set correctly, you might be unable to read the file line by line. To troubleshoot this issue, check the file permissions using the ls -l command. Then, you might need to use the chmod command to change the file permissions so that you can have the necessary access.

For example, if you are getting a “Permission Denied” error, use the chmod +r filename command to add read permission to the file.

file permissions are not set correctly

2. Blank Lines or Extra Spaces

The file being read might sometimes contain blank lines or extra spaces at the beginning or end of each line. This can cause issues when trying to process the file line by line. To troubleshoot this issue, try using the sed command to remove any extra spaces or blank lines.

For example, the command sed -e 's/^[ \t]*//' sample.txt | cat -A can remove leading spaces from each line in the file. The sed command should have removed all leading spaces or tabs from each line in the sample.txt file, and the cat -A command will show you that result.

contain blank lines or extra spaces

3. Reading Too Large File

If the file being read is very large, it can slow down the script or cause it to crash. To troubleshoot this issue, you should consider using a method that reads the file in chunks or a command, like a head command, to read only the first few lines of the file.

For example, the command head -n 5 filename can be used to read only the first 5 lines of the filename.

file being read is very large

3 Best Practices to Read Files Line by Line in Bash

To read files line by line in Bash, there are some best practices to remember to get the most out of it. Here are the top three best practices that you should follow:

  • 👀 Always check for errors: It’s important to check for errors and handle them appropriately when reading or writing files. For example, if a file doesn’t exist or can’t be opened, your script should handle the error technically. To check for errors, you can use the if statement in Bash and the test command to check if a file exists before attempting to read it. 
  • 📂 Use absolute file paths: When specifying file paths in your scripts, using them instead of relative paths is a good idea. This can help avoid confusion and ensure your script always accesses the correct file. To specify an absolute path, start the path with a forward slash (/) to indicate the root directory. You can also use commands like realpath or readlink to get the absolute file path, use it to perform your file-related operations.
  • Avoid using wildcard characters: When specifying file names in your scripts, it’s best to avoid using wildcard characters such as * and ?. These characters can cause unexpected behavior if the file name contains special characters. Instead, use a specific file name or path to ensure you access the correct file. 

To Sum Up

In this article, I’ve shown you step-by-step methods to read files line by line in bash, like using while loops and the read command. I’ve also covered common errors and best practices for smooth, error-free file processing.

If you want to dive deeper into bash scripting, check out these articles:

Frequently Asked Questions

Can I use these methods to read files in other languages?

Yes, you can use the methods explained in my guide to read files in other languages. They should work just fine as long as the file is encoded as text. However, it’s important to remember that certain languages may use different characters or character sets that could cause issues when processing the file. For example, if a file is written in a language that uses non-Latin characters, such as Chinese or Arabic, then it may require additional processing or special handling to ensure that the text is properly interpreted.

Is it possible to read a file backward?

There are various ways to read a file backward in Bash. One common method is to use the tac command, which prints out a file in reverse order. Here’s an example tac file.txt.Another way is to use the tail command with the -r option, which prints the last n lines of a file in reverse order. For example, to print the last 10 lines of a file in reverse order tail -r file.txt | head -n 10. However, these methods may not work efficiently for very large files.

Are there any security risks associated with using these methods to read files?

Yes, there is a risk that sensitive information contained in a file could be exposed if an unauthorized user reads the file or if the file is located in a directory that is accessible to unauthorized users. Always be sure to secure your files and directories properly and only grant access to authorized users.

Ojash

Author

Ojash is a skilled Linux expert and tech writer with over a decade of experience. He has extensive knowledge of Linux's file system, command-line interface, and software installations. Ojash is also an expert in shell scripting and automation, with experience in Bash, Python, and Perl. He has published numerous articles on Linux in various online publications, making him a valuable resource for both seasoned Linux users and beginners. Ojash is also an active member of the Linux community and participates in Linux forums.

Akshat

Reviewer

Akshat is a software engineer, product designer and the co-founder of Scrutify. He's an experienced Linux professional and the senior editor of this blog. He is also an open-source contributor to many projects on Github and has written several technical guides on Linux. Apart from that, he’s also actively sharing his ideas and tutorials on Medium and Attirer. As the editor of this blog, Akshat brings his wealth of knowledge and experience to provide readers with valuable insights and advice on a wide range of Linux-related topics.

Share this article
Shareable URL
Prev Post

How to Make a File Executable in Linux [3 Best Ways]

Next Post

How to Convert String to Number in Bash [6 Best Methods]

Read next