The message text is still sent to the push notification server from the app's infrastructure - this setting simply stops the phone from displaying the message.

The app itself must choose not to send the message text in the push notification.