Dilemma – What to do?

If you have read any of my posts, you know I am a cuckold, and my wife’s bull is her ex-husband. The dynamics of this relationship have evolved over time as I guess any long term relationship would. I like him, very much, and have accepted my role as the cuckolded husband, well as best I can. I know I still have to fully accept my role and learn to just enjoy it. Yes I enjoy it, I can admit to that.
Anyway, no more about me. Again if you have read my posts, you know that her ex has been spending the holidays with us. It seemed a simple, one time request, to have him celebrate the holiday here, since his kids were scheduled to be with us. No shuffling the kids between the parents. He could be here with his kids. It turned out to be a very nice time and since he spent the night, my wife and her ex could fuck. And fuck they did, but I digress. The holiday thing has become a thing, with him here now for all major holidays. I know their kids are happy to have Mom and Dad together with them, and they have accepted me as part of that dynamics, as I am Pop to them.
The Dilemma:
When my wife and her ex divorced, she moved out during the separation. He eventually bought out her half of the house after we married. It is a big house, meant for a family, just like my house. But it is too big for him. And with the joint sharing of the kids, half the time he is the only one living in it. So he is thinking of selling.

Our house is large, 5 bedrooms, the master, my son’s room when he is here, my daughter’s room, the room shared by their two boys and the guest bedroom. Her ex has slept in that guest bedroom on several occasion when his kids are with us. Of course he is in the marital bed when we have no kids in the house.
We have a basement, somewhat finished with a full bathroom.

He has suggested paying to have that basement, which we rarely used, into a separate apartment for him to move into when he sells the house. Believe me he can well afford to spend money on something that benefits us in the end when and if we sell our house. He would have his own entrance and only when he was with us would we even know he was down there. When you think of a basement, you may get the impression of a space basically underground, but our house sits on land that slops down, so the back half of that basement is like a regular house, with big windows.

Of course, if he is living downstairs, more opportunities for fucking between my wife and her ex. Also, of course, their kids would always be here. I do like those times when there are no kids in the house.

My wife thinks it may be a good idea. That we could try and see how it works. She hates the kids not being with her. He said he would not stay if it became uncomfortable for me and that he did not care about spending his money if he needed to leave. (I guess I should get that in writing.)

So it seems everything is in my court as they say. Do I want him living here full time, even though he would technically be in his own space? The times he has spend the night and shared holidays with us have been wonderful times, I have to admit. He would get to see his kids, every day. I envy that aspect but because of my situation that fucking limits my time with my kids, I do not want that for him or his kids. I know it is important for him to be in their lives.
I guess I am being too nice to him, certainly fits with my submission as a cuckold.

The plus side of course, pushed for by my sexual mind and cock, is greater opportunities for my wife to be fucked by her ex. No sneaking between guest bedroom and our marital bed when he is spending the night. There have been a couple of close calls with the kids and I do not want that for them. During the 4th of July weekend, their youngest walked into our master bedroom to find him naked getting ready to get dressed, my wife naked in bed but fortunately covered. It was one of the times I chose to let them fuck without me there, otherwise I may have prevented the kid from coming in. Explaining that was a real headache, but kids will believe a simple explanation. Fortunately they were not fucking when he entered and his Dad was not on the bed. Totally naked, with a semi-hard cock covered in cum and cunt, but fortunately the kid is too young to understand what that was all about.
With the kids getting older, we need to be more careful. So I guess him living here, but in his own space, should prevent future exposures for the kids.

Am I just trying to justify this to myself, that it is a good idea for him to move it? Is my cock overriding my common sense? I honestly do not know what is best. Any thoughts or guidance or warnings any of you may have, please let me know. This site has helped me in my journey as a cuckold, so I am hoping it can help me with this.