Hello this is an updated question to this one http://powershell.com/cs/forums/p/18455/38721.aspx
A couple months ago BOB has helped me to create this super fast code for splitting one big file into multiple small files (see link above). Unfortunately format of my data has changed so I was forced to make some changes to his code. What has changed:
1. There is no header line in the big text file anymore.
2.Names for small files are now stored in first column.
Below is updated code:
# Split-File.ps1
#
$src = "C:\Ephemeral\bigfile.csv"
$dstDir = ""C:\Ephemeral\files\""
$inData = New-Object -TypeName System.IO.StreamReader -ArgumentList $src
$currentFile = ""
while ($line = $inData.ReadLine())
{
$newFile = "$(($line -split ",")[0]).csv"
if ($newFile -ne $currentFile)
{
# We're starting on a new file
if ($currentFile -ne "")
{
# Write out contents of current file
$outData.ToString() | Out-File -FilePath $dstDir\$currentFile -Encoding ascii
}
# Get ready for a new current file
$currentFile = $newFile
$outData = New-Object -TypeName System.Text.StringBuilder
}
Write-Verbose "$currentFile,$line"
[void]$outData.Append("$($line)`r`n")
}
# Write out contents of last file
$outData.ToString() | Out-File -FilePath $dstDir\$currentFile -Encoding ascii
Problem with this code is that there is always an extra line at the end of the each split file. I've played a little bit with those "Backtick" parameters (`r`n) but I'm unable to get rid of those extra empty rows at the end of each file.
Also I would like to skip first column(column with the file names) from being copied into small files.
Example:
BIGFILE.txt
BIGFILE.TXT #NO HEADERFILE1,11/24/2013,50.67,51.22,50.67,51.12,1711/25/2013,51.34,51.91,51.09,51.87,23FILE1,12/30/2013,51.76,51.82,50.86,51.15,13FILE1,12/31/2013,51.15,51.33,50.45,50.76,18FILE1,1/1/2014,50.92,51.58,50.84,51.1,19FILE2,1/4/2014,51.39,51.46,50.95,51.21,14FILE2,1/7/2014,51.08,51.2,49.84,50.05,35FILE2,1/8/2014,50.14,50.94,50.01,50.78,100FILE3,1/11/2014,50.63,51.41,50.52,51.3,190FILE3,1/15/2014,54.03,55.74,53.69,54.93,110FILE4,1/19/2014,53.67,54.19,53.55,53.82,24FILE5,1/20/2014,53.83,54.26,53.47,53.53,23FILE5,1/21/2014,53.8,54.55,53.7,54.1,24,0FILE5,FILE1.txt#NO HEADER11/24/2013,50.67,51.22,50.67,51.12,17 11/25/2013,51.34,51.91,51.09,51.87,23 12/30/2013,51.76,51.82,50.86,51.15,13 12/31/2013,51.15,51.33,50.45,50.76,18
FILE2.txt1/1/2014,50.92,51.58,50.84,51.1,191/4/2014,51.39,51.46,50.95,51.21,141/7/2014,51.08,51.2,49.84,50.05,35
Thank you for your help.