A public space is normally defined as a place the public has access to, whither it is privately owned or not and can still be places that charge an entrance fee. It's pretty well defined in most places that restrict smoking in public places (though in the UK if it's a place that is privatley owned then you just need permission from the owner to photograph there. Permission is normally assumed, until you are told you don't have permission. But even then all they can do is eject you from the premises if you don't comply - laws may vary in other countries)
Private property rights in the U.S. vary a bit with those in Scotland and Great Britain.
Private property means just that. When you enter by invitation, you must comply with the rules of the property owner. Paid admission to an event is a limited invitation.
On public property, that is, any property owned publicly, either by a government unit, or by dedication to public use, you can generally take photos of anything in view. This is not absolute "black letter law". When you're on a public sidewalk, for instance, you can take a photo of private property, even if the owner doesn't agree (I've had this exact experience). If you attempt to photograph law enforcement, you can have problems, because the police have broad powers to interpret "interference with a police operation". Generally, if you're taking pictures of a working fire, as long as you remain out of the way of the firefighters, you're O.K.
In a shopping mall, for example, the owners may ban any photography except upon their express permission. I found this out the easy way, when a couple wanted a wedding candid at a beautiful fountain in a mall atrium. I got permission to take the photo, expressly approved by the mall manager, and limited to that location only.
If, for instance, a flea market is taking place at a municipal parking lot, you generally have permission, unless the parking lot is leased for the event. In that case, the lessee can set the privacy rules, but they have to be specific, and in writing. They also have to comply with municipal law.
And, no, I will not delete any photograph I've taken, either digital or analog.
PS - I also protect myself from assault.