This effortpost is mostly based on relevant writings by Murray Rothbard, Hans-Hermann Hoppe, and Stephan Kinsella. I recommend reading their work
At its core, libertarianism boils down to a legal philosophy. When libertarians talk about "private property", you can object that other philosophies also protect and respect property rights. The distinctive mark of libertarianism is its rules regarding particular property assignment, i.e. who is the owner of each contestable resource and how is that determined? I alluded to earlier that it starts with self-ownership, grounded in the fact that only you have direct control over your own body. What logically follows is appropriation of unowned things in nature using the body that you control. What logically follows from that is using your body and those things to produce other goods. What logically follows from that is that only non-coercive transfers of property titles from an earlier to a later possessor (trade, gift, donation, inheritance) are legitimate.
Let's call this libertarian concept of property rights the ground rules. In a libertarian order, each and every person and institution must obey these ground rules.
Now you can use different methods to answer the question "why do humans have rights?", such as natural law philosophy, Hoppe's argumentation ethics, or religious justifications. Personally, I'm really fond of the rationalist reasoning presented by Kinsella, in which he logically demonstrates that it is physically impossible to make a valid argument that rights don't exist.
In any case, in this libertarian order, nobody has any right to obtain property by means other than initial appropriation, production, or voluntary exchange (read: if I have no right to do X, you may use violence to prevent me from doing X). That means that every person (using their property) is permitted to compete against every other person in the production of all kinds of goods and services, unlike the current situation in which the state just prohibits certain activities.
This means that the production of law and order will also happen among freely financed competing service providers and companies, just like the production of all other goods and services.
I mentioned earlier that self-defense is holy, but not the most important aspect. The current statist practice is to systematically disarm the population whereas it is predictable that the opposite tendency will occur in a libertarian order, i.e. you can expect the populace to be armed. Logic and evidence show that more guns means less crime. If there is demand, I can do some additional effortposting on the philosophical foundation of gun rights, including points that are pretty much never made by gun right supporters.
With the division of labor, especially the more property you own or the wealthier society is, it is predictable that the bulk of the supply of defense and security services will come from specialized private police, insurance, and arbitration agencies.
The core difference between such a system and the current statist system can be summed up with "contract". The state essentially operates in a contractless vacuum. There is no contract between the citizen and the state. It's not set who owns what and what property is supposed to be protected. It's not set what service the state is to perform, what happens in case of an unsatisfactory performance by the state, nor what the price is that the "customer" has to pay for that "service". What happens is that the state unilaterally sets the rules of the game and changes them mid-play using legislation.
Imagine if I was a security provider, doesn't matter if I'm police or insurer or arbitrator, and I tell you the following:
I will make you no contractual guarantees at all. I am not going to tell you which exact things I'm considering protecting as "your property", nor am I going to tell you what my duty is if you think I'm not performing my service - but I definitely reserve the right to unilaterally determine my price for my undefined service.
Surely nobody would voluntarily pay for such a service.
Instead, freely financed security providers are expected to offer a contract to prospective customers. To be acceptable for voluntarily paying customers, these contracts must include clear property descriptions and clearly defined mutual duties, and they can only be changed with mutual consent by all parties. These contracts must also include what's going to happen in case of a conflict between the customer and the insurer, and in the case of a conflict between different insurers and their clients. It is expected that the parties agree to a trustworthy independent third party for arbitration.
These arbitrators are also freely financed and competing with other arbitrators and arbitration agencies. Their clients (insurers and the insured) expect them to make judgments that are considered fair by all parties. Only arbitrators who are able to make such judgments will be able to prevail in that market.
Consider fighting crime. Security providers (insurers) have to indemnify their clients in case of damages, or else they simply won't get any clients to begin with. Thus, they must be efficient in preventing crime, because if they fail to do so, they have to foot the bill. And they must be especially efficient in apprehending culprits, because only then they can make
them pay for restitution and reduce their own costs. Compare this to the current statist practice in which police get their payment irrespective of their performance. Why bother working when you get paid for doing nothing? And the victim and restitution play no role whatsoever. Instead, the victim gets fucked over again because they, as taxpayers, foot the bill to financially support the incarcerated culprit (if they ever get apprehended in the first place).
It is predictable that no insurer will be willing to cover the risk of damages following from acts of provocation. Thus, it's likely that insured persons agree to refrain from provocations of all kinds. Also, it's likely that insured persons agree to abstain from all kinds of vigilantism (except maybe in very exceptional cases). The reasoning is that vigilantism creates insecurities and provokes retaliatory acts from third parties. In agreeing to follow orderly and transparent procedures whenever they feel wronged, such disturbances and corresponding costs can be avoided.
The persecution of victimless "crimes" will hardly play a role under libertarian conditions, and that can be proven logically. Unlike real crimes against property, these "crimes" doesn't create any victims. An "insurance" against such "crimes" would thus demand higher premiums. Hardly anybody would voluntarily pay more money for such a "protection", and it's likely that freely financed insurers would simply ignore victimless "crimes" as mere private matters.
Regarding self-defense, imagine if an insurer would tell you that, before he even thinks of protecting your property, you are to disarm yourself. Surely everybody would consider this a bad joke. What I expect is what you already see from some insurance companies today, such as rewarding owners of alarm systems and safes. I expect that you will get lower insurance premiums if you're armed and trained in using weapons.
Regarding the law itself, beyond the ground rules, and beyond a trend for law to become more affordable and more efficient, I predict two trends: A larger variability of the law and more harmonization of law.
[The enforcement of rights is] left to interpretation
Beyond the ground rules, I expect security providers to not just compete in terms of price and service, but also in terms of product. You'll probably have catholic producers providing canonical law, sandnigger producers providing islamic law, nonreligious producers providing secular law, etc. This is great because it means that you don't have to live under a "foreign" law.
Essentially, it means that, for example, canonical law only gets used to resolve intra-catholic conflicts. However, of course, a catholic can get into a conflict with a mudslime. If both systems of law come to the same, or sufficiently similar, conclusion, there won't be any problem. If, however, that doesn't happen, the "domestic" law becomes useless. However, everyone wants to be insured in such a case. So we go back to what I said earlier, namely that every insurer and every one of their clients contractually agrees to the judgment of a pre-selected arbitrator, who is not just independent, but also the mutual choice of both insurers. The arbitrator is selected because he is expected to make mutually acceptable judgments in cases of inter-group conflicts. If he fails to do that, guess what, you're fired.
So, from this constant cooperation of different insurers and independent arbitrators in resolving inter-group conflicts, I expect a tendency towards a harmonization of property law and contract law, as well as procedural law, evidence law, and arbitration law. Every insured person and insurer becomes a participant in an integrated system of comprehensive conflict avoidance and peacekeeping. Every conflict and every damage claim, regardless of where, between whom, and by whom against whom, falls within the jurisdiction of one or more precisely identifiable insurers, and it will be solved either using the "domestic" law of the individual insurer, or using the "international" arbitration law which was previously contractually agreed upon.
Enough effortposting for now, I hope this is informative