I wouldn't jump to the conclusion that your ex simply "brushed it off". It is very possible that he chose to deal with the situation in his own time, in his own manner. When it is his time with his son you need to respect his parenting style. You are also expecting a lot from him. He only sees him a fraction of the time you have with him so it is reasonable that he wants the visit to be a positive, fun experience. This does not mean that he should not be a disciplinarian, just that you can't jump to conclusions about how an issue is dealt with - you weren't there for the rest of the visit.
Another issue may be that he didn't know how to discipline him. If you have been doing the majority of the parenting up until now, it is very possible that he really didn't know what to say to him, or how to deal with the problem.
How you and your ex deal with each other is going to deeply affect the development of your son. A few posts already express the same sentiments and they are correct - I have experienced it first hand. You and your ex, perhaps should begin therapy together so you can both learn to respect the parenting style of the other, but get on the same page about what is important to both of you in the raising of your son - what is punishable and what are appropriate repercussions for certain behaviours. With a therapist that has a focus on children, you will both benefit by being introduced to non-violent, respectful ways to modify behaviour.
You are not going to like everything that your ex suggests, nor is he going to like everything about your style. But, if you can meet half way and keep the major issues for both households the same you are both giving your son a continuity that only can benefit him.
You may just find that you can learn a thing or two from your ex.
Too many custodial parents attempt to continue controlling their ex through the children. Make sure that you are not doing this - even inadvertently.
Ask your ex to attend meetings with the teacher so that the three of you can develop a plan to help your son. After all the triad of parents and teachers (and often caregiver) is the most important one in your sons life right now. He should be just as invested in the well being and raising of your son as you are.
I wish you and your family the best.