Filter out users accounts based on a given csv file

I have drafted a script that will give a list of Active users from a given location but should also neglect the user accounts provided in a CSV file. I have used SAMAccountName attribute only in csv file to avoid those users. The script is as follows:

import-module activedirectory

$ADUserProperties = @(`

"DisplayName",

"sAMAccountName",

"Mail"

)

$SelectADUserProperties = @(`

"DisplayName",

"sAMAccountName",

"Mail"

)

#This will import a list of users with just their sAMAccountName provided in a csv file which has header name as 'LogonName'

$users=Import-Csv -Path C:\users\admgupta\TestActive.csv

#This should give all active users from the given location but should avoid the users given in csv 

Get-ADUser -searchbase “OU=Users,DC=contoso,DC=com” -Filter {(Enabled -eq $True) -and (samAccountName -ne '$($users.LogonName)')} -property $ADUserProperties| `

select $SelectADUserProperties |export-csv "C:\Users\admgupta\TestActiveUsers.csv" -NoTypeInformation -Encoding Unicode

This code is running but it is not eliminating the users given in CSV file.  Kindly help me out on this. Thanks in Advance.