Can I delete the backup from Google Drive?

Yes, but without a backup, the only way to access your wallet from a new device will be through the one you are currently using. Before deleting it, make sure you have carefully evaluated the consequences.