TSCMD162 FAQ: Finding duplicate filenames in a directory tree. Assorted NT/2000/XP/.. CMD.EXE script tricks written by Timo Salmi

162} Finding duplicate filenames in a directory tree.

See the responses to this posting (if the Google message link fails try the link within the brackets)

Google Groups Dec 3 2007, 2:04 am [M]
in the thread
Finding duplicate files in a directory tree Google Groups

COMPFOLD.CMD SAMENAME.CMD

Below is the solution which I use myself. It is unrelated to the references in the above. This is a somewhat complicated a task which also is heavy on computing power. Therefore, my solution depends on a UNIX uniq.exe port of common GNU utilities to native Win32 collection UnxUpdates.zip. On my system I have renamed uniq.exe to unxuniq.exe

@echo off & setlocal enableextensions
rem C:\_F\XTOOLS\FINDDUPL.CMD

:: Obviously, for debugging
set debug_=true
set debug_=

call :ProgramTitle

if "%~1"==""   (call :ProgramHelp & goto _out)
if "%~1"=="?"  (call :ProgramHelp & goto _out)
if "%~1"=="/?" (call :ProgramHelp & goto _out)

:: Assign a value for the temporary folder variable temp_
call :AssignTemp temp_
if defined debug_ (call :ShowVarValue temp_& goto _out)

:: Check that an essential auxiliary program is available
call :IsProgramAtPath "unxuniq.exe" uniqfound_
if not defined uniqfound_ (
echo The UNIX-like utility uniq.exe from UnxUpdates.zip is not avalable
echo Or has NOT been renamed unxuniq.exe as required by this script
echo For UnxUpdates.zip see e.g. http://www.elisanet.fi/tsalmi/info/tscmd.html#gawk
goto :_out)

:: Preset options of the starting root for the search, easy to customize
set switchok_=
if /i "%~1"=="/1" (
set switchok_=true
set topfolder_=%USERPROFILE%\My Documents\My Texts)
::
if /i "%~1"=="/2" (
set switchok_=true
set topfolder_=%USERPROFILE%\My Documents\My Music)

:: Check the switch's feasibility
if not defined switchok_ (
echo Error: Invalid parameter "%~1"
echo.
call :ProgramHelp
goto :_out)

:: Check for the starting point folder's existence
if not exist "%topfolder_%\" (
echo Exiting: The assigned folder to start the search from not found
echo %topfolder_%
goto :_out)

:: Remove possible leftover auxiliary files from previous runs
call :CleanUp

:: Search for the duplicates
for /f "usebackq delims=" %%f in (`
dir /s /b /a:-d "%topfolder_%\*.*"`) do (
echo %%~nxf) >> "%temp_%\tmp1.lst"
sort "%temp_%\tmp1.lst" > "%temp_%\tmp2.lst"
rem The -d switch: only print duplicate lines
rem The -i switch: ignore differences in case when comparing
unxuniq -d -i "%temp_%\tmp2.lst" > "%temp_%\tmp3.lst"
::
echo The results for %pcid_%
echo.
::
call :TestForZeroFile "%temp_%\tmp3.lst" zero_
if defined zero_ (
echo No duplicate filenames found
goto _out)
::
for /f "usebackq delims=" %%a in ("%temp_%\tmp3.lst"
) do dir /s /b "%topfolder_%\%%~a" >> "%temp_%\tmp4.lst"
for /f "usebackq delims=" %%f in (`type "%temp_%\tmp4.lst"`
) do echo "%%~f"

if not defined debug_ call :CleanUp

:: The main program ends
:_out
if not defined cmdbox if defined PauseIfFromDesktop pause
endlocal & goto :EOF

:ProgramTitle
echo +-----------------------------------------------------------+
echo ^¦ FINDDUPL.CMD Find duplicate filenames on a directory tree ^¦
echo ^¦ By Prof. Timo Salmi, Last modified Fri 9-Nov-2012         ^¦
echo +-----------------------------------------------------------+
echo.
goto :EOF

:ProgramHelp
echo Usage: FINDDUPL.CMD [StartFromFolder]
echo /1 = %USERPROFILE%\My Documents\My Texts
echo /2 = %USERPROFILE%\My Documents\My Music
goto :EOF

:IsProgramAtPath SearchFor found_
setlocal enableextensions disabledelayedexpansion
set found_=
for %%f in ("%~1") do set found_="%%~$PATH:f"
if exist "%found_%\" set found_=
if exist "%~1" if not exist "%~1\" set found_="%~1"
if [%found_%]==[""] set found_=
endlocal & set "%~2=%found_%" & goto :EOF

:AssignTemp
setlocal
set return_=%temp%
if defined mytemp if exist "%mytemp%\" set return_=%mytemp%
endlocal & set "%1=%return_%" & goto :EOF

:: For debugging, show verbally the value of an environment variable
:ShowVarValue TheNameOfTheVariable
setlocal
set par1_=%~1
call set value_=%%%par1_%%%
echo The value of %~1 is %value_%
endlocal & goto :EOF

:TestForZeroFile
setlocal enableextensions
set file_=%~1
for %%f in ("%file_%") do set size_=%%~zf
set return_=
if %size_% EQU 0 set return_=true
endlocal & set "%~2=%return_%" & goto :EOF

:CleanUp
for %%f in (
"%temp_%\tmp1.lst"
"%temp_%\tmp2.lst"
"%temp_%\tmp3.lst"
"%temp_%\tmp4.lst") do (
if exist "%%~f" del "%%~f")
goto :EOF

The question naturally arises whether the task can be done without resorting to the uniq.exe UNIX port. Yes, but the solution is more sensitive to the potential irregularities in the file names. In particular, there should be no exclamation marks (!) in the file names. The pure solution is to substitute the line

unxuniq -d -i "%temp_%\tmp2.lst" > "%temp_%\tmp3.lst"

with

call :Uniq_id "%temp_%\tmp2.lst" > "%temp_%\tmp3.lst"

and add the following subroutine to the script

:: Detect and display adjacent duplicate lines of a file
:Uniq_id
setlocal enableextensions enabledelayedexpansion
set prev_=
for /f "usebackq delims=" %%a in (`type "%~1"`) do (
set curr_=%%~a
if /i !curr_!==!prev_! echo !curr_!
set prev_=!curr_!)
endlocal & goto :EOF

References/Comments:
About a potential Vista problem Google Groups 09 Nov 2012 20:26:54 +0800

Assorted NT/2000/XP/.. CMD.EXE Script Tricks From the html version of the tscmd.zip 1cmdfaq.txt file To the Description and the Index

Assorted NT/2000/XP/.. CMD.EXE Script Tricks
From the html version of the tscmd.zip 1cmdfaq.txt file
To the Description and the Index