Is that "with you" really necessary?
I also wonder if "I told Mr. Abad that you should take me to your building (with you) on a Sunday." wouldn't sound better.
Explanation: If you leave off the "with you", the other person or people could be taking the speaker to the building and leaving him/her there. "With you", which is in the original, clarifies that building owner will be there for the transportation and the building viewing.
Where we place the prepositional phrase "on a Sunday" is unimportant. It could even be "I told Mr. Abad that on a Sunday you should take me with you to your building." But I think the way I wrote it is better than that, and the way you wrote it is equivalent.