How to select entities not within a given cylinder in minecraft

I don't think this is exactly possible. The three types of area related target selector arguments are x/y/z, dx/dy/dz and r/rm. x/y/z moves the starting point, dx/dy/dz form a cuboid and r/rm forms a sphere. If you combine the two, you can only select the cross area of the two. This can never form a cylinder.

But you can approximate it. You want to target players flying out anyway, so with the maximum speed in Spectator you won't detect them at exactly the border anyway (because command blocks/functions are restricted to 20/s and you can move up to 4 blocks in that time in Spectator mode. So you don't need to be perfectly exact with it anyway. Here are two solutions for an approximation:

Solution 1, approximating the mantle: You can put armor stands along the center axis of the cylinder (every 10 blocks should be enough), then give every player inside a radius around them a tag and then teleport everyone without the tag. Don't forget to remove the tag from everyone (*) afterwards! This makes your space look a bit like a pyramid cake (without the hole in the middle):

Solution 2, approximating the area: You can use multiple rectangles to approximate the circle. You can either do it like it's done in limit calculations:

or use the rectangles you get here (drag the point), that would probably lower the amount of rectangles you need for the same accuracy. Again, you would tag everyone inside with dx/dy/dz (y=0,dy=256), then teleport everyone without a tag and then remove the tag. This way your area would look like when someone builds a circle in Minecraft out of blocks, but you can choose the resolution.

To expand on Fabian's first solution i bring example of how to implement it.

First of we need to setup a scoreboard objective

/scoreboard objectives add inCylinder dummy inCylinder

By using this command we create a scoreboard objective InCylinder that can hold any number,

Then we summon 1 armor stand with coordinate 0 on y axis, so

/summon armor_stand 413 0 456 {NoGravity:1b}

413 is the x coordinate of the center of the cylinder,

456 is the z coordinate of the center of the cylinder,

0 shouldn't be changed

next step is to summon armor stands recursively above the original armor stand, Backup your world before doing this step

/execute @e[type=armor_stand,c=-1] ~ ~ ~ summon armor_stand ~ ~1 ~ {NoGravity:1b}

you put this in a repeating command block and let it run for ~14 seconds to summon armor stands up to y=256 because you can't build above 256 it would be useless to summon armor stands

now for the actual detecting

execute @e[type=armor_stand] ~ ~ ~ execute @a[r={SPECIFY_RADIUS_HERE}] ~ ~ ~ scoreboard players set @s inCylinder 2

put the above command into a repeating command block, and set it to always active, then connect it to a chain command block (always active)

execute @a ~ ~ ~ scoreboard players remove @s inCylinder 1

then the last chain command block does

execute @a[score_inCylinder=0] ~ ~ ~ {YOUR_COMMAND_HERE}

A really simple way you could do this without really any commands is with barriers (invisible blocks). I'm not sure this would be the way you would want to do it if you want them to be able to leave this area later but you could just use

/give (username) barrier

to get a barrier and then build them around that area.

There's a possibility that I didn't even consider before for some reason. It should even work in 1.12.2.

Summon a dummy entity at a known height and the center of the circle.

/summon armor_stand ~ 0 ~ {Marker:1,Invisible:1,NoGravity:1,NoAI:1,Tags:[""]}

Now you can just start the command execution at every player, shift it to the known height of the armour stand, check if the armour stand is in a range and, if necessary, shift the execution position back to the player:

execute at @p positioned ~ 0 ~ if entity @e[type=armor_stand,distance=..10] run <command>

It works just as easily for outside the circle (at least in 1.13+, otherwise you have to use temporary tags):

execute at @p positioned ~ 0 ~ unless entity @e[type=armor_stand,distance=..10] run <command>

This works for 1.13+.

Set up some scoreboard objectives. These will be used by every player.

scoreboard objectives add PosX dummy
scoreboard objectives add PosZ dummy
scoreboard objectives add PosXSquared dummy
scoreboard objectives add PosZSquared dummy
scoreboard objectives add Distance dummy

Store every player's X and Z position in their PosX and PosZ scores. Scale it up by 100 so you get some degree of precision (this lets you detect their position up to 1/100 of a block)

execute as @a store result score @s PosX run data get entity @s Pos[0] 100
execute as @a store result score @s PosZ run data get entity @s Pos[2] 100

Copy PosX and PosZ to PosXSquared and PosZSquared and square them

execute as @a run scoreboard players operation @s PosXSquared = @s PosX
execute as @a run scoreboard players operation @s PosXSquared *= @s PosXSquared
execute as @a run scoreboard players operation @s PosZSquared = @s PosZ
execute as @a run scoreboard players operation @s PosZSquared *= @s PosZSquared

Copy PosXSquared to Distance, then sum that with PosZSquared

execute as @a run scoreboard players operation @s Distance = @s PosXSquared
execute as @a run scoreboard players operation @s Distance += @s PosZSquared

Distance should be equal to (x^2+z^2) now. So if you wanted to test if it's between 80 and 85, you would target

@a[scores={Distance=64000000..72250000}] 

since we measured x and z in units of 1/100ths of a block so we also had to multiply by 100 (80 * 100 = 8000) and the distance hasn't been square rooted so we instead need to square our ranges (8000 ^ 2 = 64000000)

This is actually a quite janky solution since it's highly prone to integer overflow, but near the origin (such as within 85 blocks) it should work fine. If you want, you could lower the precision. If you want it to be centered somewhere else, you'd have to do a bit more math to offset PosX and PosZ before the calculation.

A way to circumvent the integer overflow would be to have a normal distance parameter that makes a sphere that encompasses the cylinder and ignore anyone outside of it. I'm sure there's got to be another cleaner solution that doesn't always result in such large numbers.

You could also simplify the process if you aren't going to use any of these scoreboard values for anything else by skipping over a few of the steps where we copy the values onto another scoreboard objective by just performing the next steps on the scoreboard objective the value is already on.