The uniq Command In Linux – Tutorial and Examples

Linux uniq Command

This article will explain how to use the uniq command in Linux to find or filter repeated lines in files and provide some usage examples.

The uniq command is a simple command which either outputs or omits repeated lines in the supplied input or file.

uniq Command Syntax

The syntax for the uniq command is as follows:

uniq OPTIONS INPUT OUTPUT

Note that:

  • OPTIONS is a list of options from the below table
  • INPUT should be the path to the file to be read
    • *Standard input *can also be used
  • OUTPUT should be the file in which the results of the uniq command are saved
    • If not supplied, standard output will be used (by default, this prints the results to the console)

If input or output files are not defined, you can redirect input and output to and from the uniq command using STDIN and STDOUT – check out this article for more info on how to do that.

Options

The below table lists the commonly used options from the user manual:

-c Prefix lines by the number of occurrences
-d Only print duplicate lines, one for each group
-D Print all duplicate lines
-i Ignore differences in case when comparing
-s Avoid comparing the first N characters
-u Only print unique lines

You can view the full user manual for the uniq command by running:

man uniq

uniq Command Examples

Example File

The following examples will use the following example file as their input:

movies.txt

Dirty Harry
Dirty Harry
Total Recall
Bride of Frankenstein
Total Recall
Demolition Man

Default Behaviour – Removing Repeated Lines

By default, the uniq command will remove repeated lines.

uniq movies.txt

Which will output:

Dirty Harry
Total Recall
Bride of Frankenstein
Total Recall
Demolition Man

Note that repeated lines are removed. Not duplicates. Only adjacent, identical lines are removed.

Showing Number of Repeats

The -c option will prepend each line with the number of times it was repeated:

uniq -c movies.txt

Which outputs:

2 Dirty Harry
1 Total Recall
1 Bride of Frankenstein
1 Total Recall
1 Demolition Man

Show Only Lines which are NOT Repeated

The -u option will show only lines which are not repeated:

uniq -u movies.txt

Which will output:

Total Recall
Bride of Frankenstein
Total Recall
Demolition Man

Show Only Lines Which ARE Repeated

The -d option will show only repeated lines:

uniq -d movies.txt

Which will output:

Dirty Harry

Again – Note that repeated lines are included or excluded. Not duplicated lines. Only adjacent, identical lines are processed by the uniq command!

Saving the Result

Pass a filename as the final parameter to the command to save the results to that file:

uniq -d movies.txt duplicated.txt

Using Standard Redirection

As mentioned above, it is not necessary to supply input and output files to the uniq command.

Standard Redirection can direct the output of other commands and applications to uniq, which can then have its own output redirected to another program or file.

cat movies.txt | uniq > output.txt

In the above example, the cat command is used to read the example file – and the output of that is then piped to the unique command. The uniq command then has its output redirected to a file using the > operator.

Obviously, this doesn’t really provide any extra functionality to just providing the paths to input/output files. Still, it’s a clear example of how the output from commands can be piped/redirected to others.

SHARE:
nv-author-image

Brad Morton

I'm Brad, and I'm nearing 20 years of experience with Linux. I've worked in just about every IT role there is before taking the leap into software development. Currently, I'm building desktop and web-based solutions with NodeJS and PHP hosted on Linux infrastructure. Visit my blog or find me on Twitter to see what I'm up to.

Leave a Reply

Your email address will not be published. Required fields are marked *