Uniq Command

Linux uniq Command Examples

Uniq Command


The uniq command is used for displaying or removing duplicate entries within files. The uniq command is often used with the sort command. If the uniq command is used alone, then it will display only unique lines and remove the duplicates. Below are some of the common uses of the uniq command along with its syntax:



Basic Syntax of command: uniq [OPTION]

The following text file "file1" will be used in the examples that follow. The contents of "file1" can be seen below:


AIX
AIX
BSD
BSD
BSD
LINUX
LINUX
OS2
PUPPY
RHEL
RHEL
SOLARIS
Zos
Zos

uniq - No Options passed



john@john-desktop:~/uniq_test$ uniq file1
AIX
BSD
LINUX
OS2
PUPPY
RHEL
SOLARIS
Zos

In the above example, the uniq command was used without any options. This resulted in any adjacent duplicates to be removed. It is important to note that the uniq command works with only adjacent duplicate entries. To illustrate this further, a file called "file2" has been created with the following lines:



AIX
BSD
AIX
BSD
AIX
BSD
LINUX
LINUX

Although the file contains duplicate entries, only adjacent duplicate entries will be removed. See example of this below:



john@john-desktop:~/uniq_test$ uniq file2
AIX
BSD
AIX
BSD
AIX
BSD
LINUX

To remove all the duplicate entries, we would have to use the "sort" command as follows:



john@john-desktop:~/uniq_test$ sort file2 | uniq
AIX
BSD
LINUX

As an alternative to the above, you can issue a "sort" command with the "-u" parameter to achieve the same results:



john@john-desktop:~/uniq_test$ sort -u file2
AIX
BSD
LINUX


uniq - Display Duplicate entries only


To display only duplicate entries in a file we can use the "-d" option with the uniq command:



john@john-desktop:~/uniq_test$ uniq -d file1
AIX
BSD
LINUX
RHEL
Zos

uniq - Display number of occurrences within a file



john@john-desktop:~/uniq_test$ uniq -c file1
      2 AIX
      3 BSD
      2 LINUX
      1 OS2
      1 PUPPY
      2 RHEL
      1 SOLARIS
      2 Zos

Using the "-c" option with the uniq command we can quickly see a count of the number of occurrences found within the file.


uniq - Display only unique entries in file



john@john-desktop:~/uniq_test$ uniq -u file1
OS2
PUPPY
SOLARIS

uniq command options



       -c, --count
              prefix lines by the number of occurrences

       -d, --repeated
              only print duplicate lines

       -D, --all-repeated[=delimit-method]
              print              all              duplicate              lines
              delimit-method={none(default),prepend,separate}  Delimiting   is
              done with blank lines

       -f, --skip-fields=N
              avoid comparing the first N fields

       -i, --ignore-case
              ignore differences in case when comparing

       -s, --skip-chars=N
              avoid comparing the first N characters

       -u, --unique
              only print unique lines

       -z, --zero-terminated
              end lines with 0 byte, not newline

       -w, --check-chars=N
              compare no more than N characters in lines

       --help display this help and exit

       --version
              output version information and exit