While you may have SSH access disabled, remote vCLI access and PowerCLI access is still possible unless lockdown mode is enabled.
If you enable lockdown mode, all remote management of the ESXi hosts (whether using vSphere Client, vCLI/vMA, or PowerCLI) must first be connected through vSphere Server.
Hope this helps.