How do I correctly setup caching for my custom api routing?

I have custom module with some routes which return json, for instance terms of a vocab.

routing.yml

mymodule.terms:
  path: '/api/terms/{vid}'
  defaults:
    _controller: 'DrupalmymoduleControllerMyModuleController::getTerms'
    _title: 'Terms'
  requirements:
    _permission: 'access content'
    vid: '^(foobar|barfoo)$'

Controller:

public function getTerms($vid) {
  $response = $this->jsonifyVocabulary($vid);
  return $response;
}

private function jsonifyVocabulary($vid) {
  $request = Drupal::request();
  $lang = $request->query->get('lang', 'en');
  $output = [];
  $taxonomyManager = Drupal::entityManager()->getStorage('taxonomy_term');
  $terms = $taxonomyManager->loadTree($vid, 0, 1, false);
  $response = CacheableJsonResponse::create($output);

  debug(1);

  foreach ($terms as $term) {
    $term = Term::load($term->tid);
    if ($term->hasTranslation($lang) && $lang !== $term->language()) {
      $term = $term->getTranslation($lang);
    }
    $entry = [
      'tid' => (int) $term->id(),
      'label' => $term->getName(),
    ];

    // $response->addCacheableDependency($term);
    $output[$vid][] = $entry;
  }

  $response->setData($output);

  $cacheMeta = (new CacheableMetadata())->addCacheContexts(['url.query_args:lang']);
  $response->addCacheableDependency($cacheMeta);

  return $response;
}

I know how to actually add the url params as a cache context, but the

debug(1);

always gets called, I guess I just somehow have to tell Drupal that this json response’s cache should relate to that route…

I dont use the core’s rest module on purpose since I find it hard to configure to output exactly what I need.

Drupal version: 


Source: https://www.drupal.org/taxonomy/term/4/feed