I agree with you. And, as we have both noted, we both know people in the LGBTQ community, whom we treat well and decently, so I doubt we are LGBTQ-phobic.
As a caveat, though, I would say that you can "harm" a person without violence: denying them a job, not promoting them, or firing them...