looking for a tool to conver line endings

[rvida]

Hi,

I am looking for a simple tool that can

1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line

[nepossiver]

To convert from dos to unix and vice-versa (and I think the package also includes the Mac text files, which if I am not mistaken is CR), there is dos2unix:

http://sourceforge.net/projects/dos2unix/

To trim space you could use a perl scrit - to do the format conversion as well, in fact.

[IGarcia]

Hi,

I am looking for a simple tool that can

1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line

The white space has to be before end-line, so inverting task order you can do both in a single step, run:

Code: Select all

perl -pe 's/ *\r\n/\n/' inputfile

It will replace all (optional) white spaces followed by CR+LF by a single LF from a file to stdout.

Hope it helps.
Regards

Ignacio

[lucasart]

Hi,

I am looking for a simple tool that can

1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line

Regular expressions can do it

First remove all trailing spaces and tabs at the end of each line with sed:

Code: Select all

sed 's/[ \t]*$//' file.txt > file_trimmed.txt

I'm not sure if sed will automatically convert CRLF into LF, but if it doesn't just pipe this into another sed

Code: Select all

sed 's/[ \t]*$//' file.txt | sed 's/\n/\n/' > file_trimmed.txt

[bob]

Hi,

I am looking for a simple tool that can

1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line

Linux has always had dos2unix and unix2dos commands. That what you want?

[nepossiver]

I am not fluent on perl one-liners, but now I am jealous of the previous posts, so here is a (untested) script to convert to / from any OS specific format:

Code: Select all

#!/usr/bin/perl
open &#40;fh_in, "<", $ARGV&#91;0&#93;) or die "could not open file $!";
open &#40;fh_out, ">", $ARGV&#91;1&#93;) or die "could not open file $!";

while (<fh_in>)
&#123;
	$_ =~ s/\r//;
	$_ =~ s/\n//;
	$_ =~ s/\s+$//;
	if ( $ARGV&#91;2&#93; =~ m/mac/i ) &#123; $_ = $_."\r"; &#125;
	elsif ( $ARGV&#91;2&#93; =~ m/unix/i ) &#123; $_ = $_."\n"; &#125;
	elsif ( $ARGV&#91;2&#93; =~ m/windows/i ) &#123; $_ = $_."\r\n"; &#125;
	print fh_out;
&#125;
close &#40;fh_in&#41;;
close &#40;fh_out&#41;;

just save this as whatever_you_like.pl, then make it executable:

Code: Select all

 chmod +x whatever_you_like.pl

and call it as:

Code: Select all

 ./whatever_you_like.pl input_file output_file SYSTEM

SYSTEM being any one choice of mac, unix or windows (case-insensitive). Again, untested, if does not work, just get back here and someone will point to any possible errors.

edit: this assumes you are developing on unix/mac, if not, you have to call the script as:

Code: Select all

perl whatever_you_like.pl input_file output_file SYSTEM

[IGarcia]

Hi,

I am looking for a simple tool that can

1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line

Linux has always had dos2unix and unix2dos commands. That what you want?

those programs are not installed by default and the other problem is not solved: you still have spaces before end-line.

@Horacio: Nice coding.
My command misses tabs.. This will do the trik form command line

Code: Select all

perl -pe 's/\s*\r\n/\n/' infile > outfile

[rvida]

Thanks for all the answers.

I decided to use the sed based solution. It is a wonderful tool, although for people coming from Dos/Windows world the syntax is somewhat obscure...

Btw. after some googling I found a list of very useful sed one liners:
http://sed.sourceforge.net/sed1line.txt

[IGarcia]

Thanks for all the answers.

I decided to use the sed based solution. It is a wonderful tool, although for people coming from Dos/Windows world the syntax is somewhat obscure...

Btw. after some googling I found a list of very useful sed one liners:
http://sed.sourceforge.net/sed1line.txt

Sure, sed is a great.

There are little, but important, differences on how each program interprets regular expressions. Keeping track of those differences in memory is a mess so its common to stick to one of few programs.

Regular expression (regex) syntax is obscure but is all logic behind them, and crafting some regex can be great fun, as solving a chess problem!


Ignacio.

[Sven]

I decided to use the sed based solution. It is a wonderful tool,

Hi Richard,

take care of doing both steps (CR/LF->LF conversion + trailing whitespace removal) separately, starting with the CR/LF part. Depending on the platform where you perform these conversions, "sed" as well as "perl" or other tools using regular expressions may or may not recognize a CR/LF character sequence as something that matches a "$" (end of input line) in the given pattern. Therefore a pattern logically resembling "<whitespace><whitespace>*$" may or may not match an input line that ends with <whitespace><CR><LF>. You can expect it to succeed in a typical Windows-like environment where CR/LF is the typical text file line ending, but not in a typical UNIX environment. Furthermore, also combining "<whitespace><whitespace>*<CR><LF>" in one pattern will not always succeed since line endings could be inconsistent within one file.

although for people coming from Dos/Windows world the syntax is somewhat obscure...

Hmmm ... wasn't it an invention from the DOS world to have that CR/LF line ending that created one of the biggest (in)compatibility issues in the whole IT world?

Sven