What are you you trying to archive with setting the limit?
You are actually forcing ballooning with setting a low memory limit. Is there a chance you are mixing up "Limit" and "Reservation"? A reservation will ensure the VM is granted a certain amount of physical memory.
With that said, unless you are low on memory, I don't recommend to set either a Limit or a Reservation at all and let the ESXi resource scheduler take care of this.
André