How can I programmatically change the view mode?

In a preprocess function, it can be done depending of a value of one field like this:

function hook_preprocess_paragraph(array &$variables) {
  $paragraph = $variables['paragraph'];
  if ($paragraph->field_paragraph_image_clickable->value === "0") {
    $variables['content']['field_paragraph_image_ref']['0']['#view_mode'] = 'image_without_link';
    unset($variables['content']['field_paragraph_image_ref']['0']['#cache']['keys']);
  }

Extending @tim's answer, I found that I had to change the cache key as well.

function HOOK_entity_view_alter(array &$build, DrupalCoreEntityEntityInterface $entity, DrupalCoreEntityDisplayEntityViewDisplayInterface $display) {
      if ($entity->getType() === 'my_content_type') {
        // Get the value of the "setting" field.
        switch ($entity->field_list_style->value) {
          case 'custom':
            // Set the view mode.
            $count = $build['field_my_entity_reference']['#items']->count();
            for ($i = 0; $i < $count; $i++) {
              $build['field_my_entity_reference'][$i]['#view_mode'] = 'custom_view_mode';

              $cache_key = array_search('default_view_mode', $build['field_my_entity_reference'][$key]['#cache']['keys']);
              $build['field_my_entity_reference'][$key]['#cache']['keys'][$cache_key] = 'custom_view_mode';
            }
            break;
        }
      }
    }

If you have an entity reference field attached to a node, and want to offer the editor a choice of view modes through another field, you can change the view mode in the render array.

In this example my_content_type has a field field_my_entity_reference that is set up to be rendered using default_view_mode. The field_list_style field offers a list of display options that are used to override the default view mode.

function HOOK_entity_view_alter(array &$build, DrupalCoreEntityEntityInterface $entity, DrupalCoreEntityDisplayEntityViewDisplayInterface $display) {
  if ($entity->getType() === 'my_content_type') {
    // Get the value of the "setting" field.
    switch ($entity->field_list_style->value) {
      case 'custom':
        // Set the view mode.
        $count = $build['field_my_entity_reference']['#items']->count();
        for ($i = 0; $i < $count; $i++) {
          $build['field_my_entity_reference'][$i]['#view_mode'] = 'custom_view_mode';
        }
        break;
    }
  }
}

You can select VIew Mode as:

function YOUR_MODULE_entity_view_mode_alter(&$view_mode, DrupalCoreEntityEntityInterface $entity, $context) {
  if ($entity->getEntityTypeId() == 'taxonomy_term' && $entity->bundle() == 'tag' && $view_mode == 'full') {
    $view_mode = 'CUSTOM VIEW MODE';
  }
}