Duplicate queries not correctly detected

[rubengrill]

With django debug toolbar 2.2 (since 2.0) I see a lot of duplicated queries.
When turning on logging for sql queries, I see them only once, and when debugging, those queries are clearly executed only once.

I found out that this only happens when using chunked cursors, because debug toolbar wraps the cursor twice, leading to logging it twice and wrongly marking it as duplicated.

Please see this line:
https://github.com/jazzband/django-debug-toolbar/blob/2.2/debug_toolbar/panels/sql/tracking.py#L58

It wraps the cursor returned from connection._djdt_chunked_cursor.
For databases without a special implementation for chunked cursors (e.g. mysql), that method actually just runs connection.cursor:
https://github.com/django/django/blob/2.2.10/django/db/backends/base/base.py#L572

Which in turn wraps the cursor a second time:
https://github.com/jazzband/django-debug-toolbar/blob/2.2/debug_toolbar/panels/sql/tracking.py#L53

[tim-schilling]

Do you have a test case that we can reproduce this with?

[valentijnscholten]

I am seeing the same, not sure how I can easily create a test case as I am seeing this in a project with a models.py which by itself is over 3000 lines...

It seems to only be happening for queries that involve django-filter to render dropdown/multiselect html widgets and consistenly show 2 queries, of which the first 1 is very fast:

Don't know if a stacktrace helps, but both queries have the exact same stacktrace:

/home/user/venv_dd/lib/python3.6/site-packages/django/contrib/staticfiles/handlers.py in __call__(65)
  return self.application(environ, start_response)
/home/user/venv_dd/lib/python3.6/site-packages/ddt_request_history/panels/request_history.py in patched_middleware_call(56)
  response = toolbar.process_request(request)
/home/user/venv_dd/lib/python3.6/site-packages/ddt_request_history/panels/request_history.py in process_request(195)
  return super().process_request(request)
/c/Data/django-DefectDojo/dojo/middleware.py in __call__(62)
  response = self.get_response(request)
/c/Data/django-DefectDojo/dojo/middleware.py in __call__(47)
  response = self.get_response(request)
/c/Data/django-DefectDojo/dojo/finding/views.py in open_findings(79)
  return findings(request, pid=pid, eid=eid, view=view, filter_name="Open", query_filter=OPEN_FINDINGS_QUERY)
/c/Data/django-DefectDojo/dojo/finding/views.py in findings(187)
  'jira_config': jira_config,
/home/user/venv_dd/lib/python3.6/site-packages/django/shortcuts.py in render(36)
  content = loader.render_to_string(template_name, context, request, using=using)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader.py in render_to_string(62)
  return template.render(context, request)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/backends/django.py in render(61)
  return self.template.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(171)
  return self._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(150)
  return compiled_parent._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(62)
  result = block.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(188)
  return template.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(173)
  return self._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/defaulttags.py in render(209)
  nodelist.append(node.render_annotated(context))
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(993)
  return render_value_in_context(output, context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_value_in_context(972)
  value = str(value)
/home/user/venv_dd/lib/python3.6/site-packages/django/utils/html.py in <lambda>(388)
  klass.__str__ = lambda self: mark_safe(klass_str(self))
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/boundfield.py in __str__(33)
  return self.as_widget()
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/boundfield.py in as_widget(93)
  renderer=self.form.renderer,
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in render(241)
  context = self.get_context(name, value, attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in get_context(678)
  context = super().get_context(name, value, attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in get_context(639)
  context['widget']['optgroups'] = self.optgroups(name, context['widget']['value'], attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in optgroups(587)
  for index, (option_value, option_label) in enumerate(self.choices):
/home/user/venv_dd/lib/python3.6/site-packages/django_filters/fields.py in __iter__(252)
  for value in iterable:
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/models.py in __iter__(1137)
  for obj in queryset:

django-auditlog==0.4.7
django-filter==2.3.0
django-multiselectfield==0.1.12
django-crispy-forms==1.9.1
django_extensions==2.2.9
Django==2.2.13
uWSGI==2.0.18
vobject==0.9.6.1
whitenoise>=4.1.2
django-cachalot==2.2.0
django-debug-toolbar==2.2

[carltongibson]

I just hit this with SQLite. I'll try and follow-up with a sample project next week, but it's just simple model.

Commenting out the overriding of the chunked_cursor (as suggested per @rubengrill's #1239 (comment)) revolves the issue.

I'm guessing a check to see if the returned cursor is already wrapped might be sufficient. 🤔

[saemideluxe]

I encountered the same problem while trying to investigate a performance issue. This is rather confusing because one sees the same query and the same stack trace but different times. Preventing the wrapping of chunked_cursor resolved the issue. I made a PR #1475 but not sure if this is acceptable like that.
Btw. I stumbled over your talk about ModelChoiceField @carltongibson due to the same issue. Thanks, good and helpful talk :)

[carltongibson]

@saemideluxe — Glad it was helpful. Thanks 😊

[jayaddison]

#1478 offers an attempt to add test coverage for this issue (including both chunked and non-chunked cursors, to try to avoid any future regressions).