Chokes will typically fall into two categories: Constriction and interference. Constriction chokes are the most common style of shotgun chokes. These type use a tapered cylinder to squeeze the shot and wad to keep the grouping of pellets closer in space. The tighter the choke, the tighter the pattern. An interference style of choke tube, like the infamous Patternmaster, use a set of small studs to grab and slow the wad, allowing the shot to freely exit the barrel in a column of packed shot. The idea is that the shot sting will be shorter and therefor there will be more pellets hitting a target at once.
Bigger is better right? Well, the answer is complicated. Bigger pellets can carry more energy because of their mass, but that comes at some costs. The most obvious being limited space in a shotshell. The less pellets in a payload require a tighter choke to keep the spacing between pellets small enough at an intended range. Check out Tom Roster's Shot Lethality Table to learn more about recommended shot and choke combinations.
The second main cost is friction, both with the air and with the object it hits. The use of heavier than steel materials for pellets has seen a spike in popularity recently. Bismuth pellets, like BOSS, carry more energy per pellet of the same size because of its density advantage. Taking it a set further, replacing steel shotshells with bismuth, you can choose a smaller shot size which typically will increase the number of pellets in a shell. For example, 3" 1.25oz 2 shot (steel) is a common load late season mallards. I recently switched over to BOSS and shoot 3" 1.5oz 4 shot (bismuth). The resulting payload has 234 pellets of bismuth compared to the 153 pellets of steel with the same effective range! The greater number of pellets allows denser patterns out of a given choke. These smaller pellets also penetrate better than steel.
If you want to do deep dive into bismuth Dave from AZ has a great article to satisfy nerds like me.
Select your test loads. Choose loads that you would like to hunt with and pellet sizes that are appropriate for the game you're chasing.
Gather targets. Many people like cardboard to pattern their shotguns but I prefer rolls of paper. You can buy a roll of masking paper from a hardware store or some auto parts stores. With the roll on the ground, I use a cardboard sheet as backing and roll up a new slice of paper between shots. Make sure the target is big enough to show your whole pattern ~40 inches
Shoot test loads. Alternate load and choke combinations at the yardage you realistically shoot at regularly. If you hunt over decoys, try something a little more open like a Improved cylinder. If you find yourself pass shooting a lot, go ahead and put in that full choke. Most hunters fall somewhere in the middle though the tendancy is overestamate how far you shoot as some sort of act of manliness.
Interpret the results. What your looking for is a fairly uniform pattern with pellets densely packed enough to humanly kill what you are shooting at. If you are unsure, refer to Tom Roster's Shot Table
Pick a shell and stick with it! Switching up loads with different velocitys can mess up your shooting more than you think. Don't believe me? Sean will back me up in this Meateater Article.