Delete all empty folders in a directory tree - oops, script deletes files too

I have a large folder that has both empty and full folders. I am using SBS 2008 and PowerShell 2.0. I used the following script (running from within the PowerShell ISE as an administrator) to recursively delete only empty folders:

dir 'P:\File Server\Cleanup\' -recurse |

Where { $_.PSIsContainer -and @(dir -Lit $_.Fullname -r | Where {!$_.PSIsContainer}).Length -eq 0 } |

Remove-Item -recurse

However, I noticed that before the script ran I had 17,832 folders and 232,343 files. After the script ran I had 12,622 folders and 201,286 files. Oops.

Furthermore, I saw the following errors in the ISE which puzzled me:

+ Remove-Item <<<<  -recurse

+ CategoryInfo: PermissionDenied: (desktop.ini:FileInfo) [Remove-Item], IOException

+ FullyQualifiedErrorId : RemoveFileSystemItemUnAuthorizedAccess,Microsoft.PowerShell.Commands.RemoveItemCommand

Remove-Item: Directory P:\path\to\a\folder cannot be removed because it  is not empty.

Wait, WHAT? It's not supposed to even get to the point of attempting to delete a folder if it's not empty!

1. Why did files get deleted when it appears that the logic should have prevented that?
2. Why are deletions being attempted on folders with files in them?