What the most you should be able to ask for a security deposit while still abiding by the law and not scaring your potential tenants away?
First and foremost you need to follow the laws in the state or province where your rental unit is located.
Most jurisdictions regulate what kind and maximum amounts of security deposit you can request. (if any) As a landlord you must know the rules in YOUR area.
Not scaring tenants away, you must be within the range of comparable properties. Call around see what other landlords are asking.
It is best to be firm but fair. If the tenant is not able or willing to provide a reasonable security deposit, then how easily can rent be collected?