I had a similar problem with my Zigbee network when I first started, but I was able to dig in and fix all of the issues. I changed the Zigbee channel and moved everything away from my WiFi router. @NathanCu was really helpful [1] on the Home Assistant community forums.

[1] https://community.home-assistant.io/t/zigbee-problems-how-to...