Absolutely not. You don’t have to believe that everyone will be good for freedom to work. America’s Founding Fathers understood that most people were influenced by greed, ambition, and the pursuit of their own happiness. That is why they built structures with limited powers, checks and balances, and other limiting factors on the darker sides of human nature. If people are generally not good, the last thing you should want is a powerful government staffed by those evil folks.
Self-government breeds individual responsibility and stronger character, which decreases violence and theft.
Most people, most of the time, deal with each other on the libertarian premise of respect for the person and property of others. The majority of people do not steal, or cheat or murder others. A strong respect for property and civil liberties gives authority to justice systems to punish those few who violate the law and provide restitution for those harmed.