I left my lover for my husband but i don't love my husband!?

i 've married for 9years. we have 2 kids. i was never happy with him. we had what my therapist calls a toxic relationship. i met a wonderful man 2 yrs ago and we became lovers. he was kind and attentive but he was also married. he didn't love his wife and wasn't happy in his marriage. had no kids. for 2 years i was hoping he'll leave her and i'll divorce my husband and we'll move in together. when this finally happened. he told me he'll tell her he wants out and that i should also tell my husband. i chickened out. told him i don't want a divorce and broke up with him. now i'm miserable. i don't know why i did that? i'm so unhappy and miss my lover. but i know that i screwed up and he'll never have me back. he's too damn proud. what should i do?