It was believed to be a bull shark, measuring around 3.5m long - local authorities have since installed signs warning people to avoid swimming in the area, and for pet owners to keep their dogs out of the water.
Nigel, from south Sydney, said he had owned Molly for two years after rescuing her from the pound.
"It was very quick".
Experts say shark attacks are increasing as water sports become more popular and bait fish move closer to shore, but human fatalities remain rare.
The dog was reported to be swimming just several meters off-shore when it was taken by the shark at Bonna Point, near the south-eastern headland of Botany Bay.
"It just took her under the water", Nigel was quoted as saying. They did not see Molly again, and could not retrieve her body.
Swimmers and dog owners were warned after Molly was taken by a 3.5m bull shark
"We just freaked out and we told a few people what we saw on the way back", Nigel told the Herald.
Apart from the shock of losing his beloved pet, Nigel said swimmers should also be aware.
The dog's owners witnessed the incident at Bonna Point, a popular off-leash beach, said Sutherland Shire Council.
A poor dog has reportedly been taken by a bull shark while indulging in an evening swim on Sunday and our hearts just can't take it.
This comes after several shark sightings in the Botany Bay area this month.
On 9 February, a kite surfer was recorded on video riding towards a shark at Brighton Le Sands. On February 14 a young great white shark was spotted in Alexandra Canal in the Cooks River.