Once you have signed a contract with the estate agent for your sale, you are legally bound to pay fees if a buyer is found. You haven't signed anything so you don't owe them any commission. The way to avoid the situation you describe with private buyers is, up front, name them on the contract that you sign as being 'exclusions' to the contract - in other words, if you sold to one of the named private buyers, no commission would be paid. This is more common when one changes estate agent and a potential buyer has been introduced by the first agent who can't yet proceed but wants the house - they would be named as an exclusion to stop the vendor having to pay BOTH estate agents for introducing the same buyer.
However the builder is going to have to pay that estate agent the commission on the full sales value on the sale of his house to you - not the lower part-exchange figure.
I guess that you also know that you can't avoid Stamp Duty Land Tax by doing this (it probably wasn't your intention anyway) - it is still assessed on both purchases at their market value.