Marry Revert Despite Family Anger?

I wanted to ask that I am a girl 28 years old I used to work in a company where I met a man who was non muslim before.We were in a haram relationship and I decided to end it because i knew that our path would make us reach hell I told him that if you want to marry me that you've to convert genuinely and practice islam not just convert for marriage purpose My brother knew about my relationship and made me understand about consequences after which I decided to end it There was 1 year gap in which I didn't talk to that man and in that duration he reverted and also learnt about islam I repented to Allah about my sin and asked allah to guide me to the straight path Now that guy approached me for marriage and I got a soft corner for him because he is revert I talked to my brother about this he is very angry saying you continued to the same path you left before this is your planning you are making us fool and giving me swears telling to leave it for the sake of parents he is saying that all men are same and he will use you and then step back to his religion but as far as I know he doesn't have bad qualities of mixing with opposite gender and he is very respectful to others .Also my brother is saying if you marry him I will cut all ties with you I will never ever support you nor talk to you till death he said I will make sure that my parents also don't agree with this marriage also they have said what will people think you married a revert he has no family whatsoever Please guide, I have done istikhara multiple times and my heart is still firm on one decision , I want to marry that man and also don't want to hurt my family