According to its Pokédex entry on Pokemon.com, "[Smoliv] protects itself from enemies by emitting oil from the fruit on its head. This oil is bitter and astringent enough to make someone flinch." We can infer that Smoliv was frightened by the Quaxly, and thus thought it was an enemy.