Right, smudges on the lens drive me nuts during my outdoor photo sessions too. I found that using a rocket blower to puff away dust first is super helpful and then following up with a microfiber cloth that's slightly moistened does the trick for those stubborn spots.
Works great. Just be gentle no harsh rubbing or you'll regret it, and always store your gear in a clean bag to prevent more buildup yeah that's my quick tip from trial and error.