iOS devices stop communicating after OS update

We have our own MDM service deployed to manage our enterprise fleet of devices. We have faced this problem more than a few times.
After OS update, a bulk of iOS devices completely lose connection with the MDM server. Investigating the device logs reveals that the pushmagic is wrong, during the MDM wake up/ MDM check in.
We know that iOS devices send a token update that contains the new token and push magic, during OS updates, APNS renewal etc. We have thoroughly analyzed our MDM service logs to ensure that the MDM service has not lost the new token or push magic, when the device sent new ones. Still the device logs that the push magic is wrong and does not contact MDM server, when notified.
We’ve tried checking about this problem with Apple support but they’re not aware of any known issues (they had some issue once during iOS 9). We have faced this a couple of times (incidentally we assume its during OS updates) and lost majority of devices contact (NOT all). Once we lost contact with bulk of Single app mode devices and we had to factory reset them all!
Has anyone faced similar problems with your MDM services? Any pointers?

What version of iOS are you updating from and to?

Once during iOS 9 to 10. A second time during a recent iOS 11 update. I could not find specific patterns with respect to os versions. Devices were on different versions before update.
I’ve narrowed down OS update to be the most probable cause. After eliminating scenarios such as server down time etc. Server down times should cause the device to report new tokens again and again.

What MDM platform are you using?

I’ve seen some issues checkin between device and MDM in a misconfigured HA scenario

You have multiple MDM servers behind a load balancer


A - Issues an APNS command and waits for device to check in
B - Device connects to server B which does not have any command queued for the device

Device checkin fails

This was caused by LB configuration and issues with how commands are handled by the MDM vendor

We have faced this issue many a times. When we look into device logs we get following error.

Aug 26 08:03:04 iPhone6 mdmd(libdispatch.dylib)[4508] : Push token received.
Aug 26 08:03:04 iPhone6 mdmd(ApplePushService)[4508] : Received push notification.
Aug 26 08:03:04 iPhone6 mdmd(ApplePushService)[4508] : Rejecting MDM push dictionary because it does not have the right magic string.

I can learn that the MDM does not have updated pushmagic thats in device. But i can confirm there was no server downtime . We suspect device have not posted token update message. We have experienced few times when there is device OS update and other times when update the MDM profile itself.

I can see few other post in in apple forums check this .