Quantcast
Channel: PowerShell.com – PowerShell Scripts, Tips, Forums, and Resources
Viewing all articles
Browse latest Browse all 6937

How to split one big text file into multiple files in powershell - update

$
0
0

Hello this is an updated question to this one http://powershell.com/cs/forums/p/18455/38721.aspx

A couple months ago BOB has helped me to create this super fast code for splitting one big file into multiple small files (see link above). Unfortunately format of my data has changed so I was forced to make some changes to his code. What has changed:

1. There is no header line in the big text file anymore.

2.Names for small files are now stored in first column.

Below is  updated code:

 

# Split-File.ps1
#
$src = "C:\Ephemeral\bigfile.csv"
$dstDir = ""C:\Ephemeral\files\""

$inData = New-Object -TypeName System.IO.StreamReader -ArgumentList $src

$currentFile = ""
while ($line = $inData.ReadLine())
{
    $newFile = "$(($line -split ",")[0]).csv"
    if ($newFile -ne $currentFile)
    {
        # We're starting on a new file
        if ($currentFile -ne "")
        {
            # Write out contents of current file
            $outData.ToString() | Out-File -FilePath $dstDir\$currentFile -Encoding ascii
        }
        # Get ready for a new current file
        $currentFile = $newFile
        $outData = New-Object -TypeName System.Text.StringBuilder
    
    }
    Write-Verbose "$currentFile,$line"
   [void]$outData.Append("$($line)`r`n")

}
# Write out contents of last file
$outData.ToString() | Out-File -FilePath $dstDir\$currentFile -Encoding ascii

Problem with this code is that there is always an extra line at the end of the each split file. I've played a little bit with those "Backtick" parameters (`r`n) but I'm unable to get rid of those extra empty rows at the end of each file.

Also I would like to skip first column(column with the file names) from being copied into small files.

Example:

BIGFILE.txt

 

BIGFILE.TXT

                                                     #NO HEADER
FILE1,11/24/2013,50.67,51.22,50.67,51.12,17FILE1,11/25/2013,51.34,51.91,51.09,51.87,23FILE1,12/30/2013,51.76,51.82,50.86,51.15,13FILE1,12/31/2013,51.15,51.33,50.45,50.76,18FILE2,1/1/2014,50.92,51.58,50.84,51.1,19FILE2,1/4/2014,51.39,51.46,50.95,51.21,14FILE2,1/7/2014,51.08,51.2,49.84,50.05,35FILE3,1/8/2014,50.14,50.94,50.01,50.78,100FILE3,1/11/2014,50.63,51.41,50.52,51.3,190FILE4,1/15/2014,54.03,55.74,53.69,54.93,110FILE5,1/19/2014,53.67,54.19,53.55,53.82,24FILE5,1/20/2014,53.83,54.26,53.47,53.53,23FILE5,1/21/2014,53.8,54.55,53.7,54.1,24,0

FILE1.txt#NO HEADER 11/24/2013,50.67,51.22,50.67,51.12,17 11/25/2013,51.34,51.91,51.09,51.87,23 12/30/2013,51.76,51.82,50.86,51.15,13 12/31/2013,51.15,51.33,50.45,50.76,18
FILE2.txt
1/1/2014,50.92,51.58,50.84,51.1,191/4/2014,51.39,51.46,50.95,51.21,141/7/2014,51.08,51.2,49.84,50.05,35


Thank you for your help.

Viewing all articles
Browse latest Browse all 6937

Trending Articles