Skip to content

Test: Raise test error for unexpected warn/error messages #4675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Test: Raise test error for unexpected warn/error messages #4675

wants to merge 1 commit into from

Conversation

marcotc
Copy link
Member

@marcotc marcotc commented May 21, 2025

Oh boy, this might take a while to fix all tests...

Change log entry
No

How to test the change?

@github-actions github-actions bot added the dev/testing Involves testing processes (e.g. RSpec) label May 21, 2025
@datadog-datadog-prod-us1
Copy link
Contributor

datadog-datadog-prod-us1 bot commented May 21, 2025 •
edited
Loading

Datadog Report

Branch report: tmp/marco-test-this
Commit report: 389084c
Test service: dd-trace-rb

❌ 833 Failed (0 Known Flaky), 18367 Passed, 1293 Skipped, 23m 18.48s Total Time
❄️ 169 New Flaky

❌ Failed Tests (833)

This report shows up to 5 failed tests.

  • ActiveRecord multi-database implementation when databases are configured with a String that's a URL an invalid value for a typical server fails to configure - rspec - Details

    Expand for error 
     Unexpected Datadog.logger.error(): Failed to resolve key "some invalid string". Cause: URI::InvalidURIError Source: /opt/jruby/lib/ruby/stdlib/uri/rfc2396_parser.rb:177:in \`split'
 
 Failure/Error: raise "Unexpected Datadog.logger.error(): #{msg}"
 
 RuntimeError:
   Unexpected Datadog.logger.error(): Failed to resolve key "some invalid string". Cause: URI::InvalidURIError Source: /opt/jruby/lib/ruby/stdlib/uri/rfc2396_parser.rb:177:in \`split'
 ./spec/support/log_helpers.rb:24:in \`block in included'
 ./lib/datadog/tracing/contrib/active_record/configuration/resolver.rb:97:in \`parse_matcher'
 ./lib/datadog/tracing/contrib/configuration/resolver.rb:51:in \`get'
 ./lib/datadog/tracing/contrib/configurable.rb:49:in \`configure'
 ...

  • contrib integration testing dynamic configuration with dynamic configuration data for tracing_header_tags changes the HTTP header tagging for span - rspec - Details

    Expand for error 
     expected 2 items, got 1
 
 Failure/Error: expect(spans).to have(2).items
   expected 2 items, got 1
 ./spec/datadog/tracing/contrib/suite/integration_spec.rb:177:in 'block (5 levels) in <top (required)>'
 ./spec/support/http_server_helpers.rb:55:in 'block in HttpServerHelpers::ClassMethods#http_server'
 ./spec/datadog/tracing/contrib/suite/integration_spec.rb:19:in 'block (3 levels) in <top (required)>'
 /usr/local/bundle/gems/climate_control-1.2.0/lib/climate_control.rb:24:in 'block in ClimateControl#modify'
 /usr/local/bundle/gems/climate_control-1.2.0/lib/climate_control.rb:15:in 'Monitor#synchronize'
 /usr/local/bundle/gems/climate_control-1.2.0/lib/climate_control.rb:15:in 'ClimateControl#modify'
 ...

  • Datadog::AppSec::Component#reconfigure when the new processor is not ready does not swap the processor instance and finalize the old processor - rspec - Details

    Expand for error 
     Unexpected Datadog.logger.error():
 
 Failure/Error: raise "Unexpected Datadog.logger.error(): #{msg}"
 
 RuntimeError:
   Unexpected Datadog.logger.error(): 
 ./spec/support/log_helpers.rb:24:in \`block in included'
 ./lib/datadog/appsec/processor.rb:77:in \`create_waf_handle'
 ./lib/datadog/appsec/processor.rb:20:in \`initialize'
 ./lib/datadog/appsec/component.rb:96:in \`block in reconfigure'
 ...

  • Datadog::AppSec::Processor::RuleLoader#load_ruleset when ruleset is a non existing path returns nil`` - rspec - Details

    Expand for error 
     Unexpected Datadog.logger.error():
 
 Failure/Error: raise "Unexpected Datadog.logger.error(): #{msg}"
 
 RuntimeError:
   Unexpected Datadog.logger.error(): 
 ./spec/support/log_helpers.rb:24:in \`block in included'
 ./lib/datadog/appsec/processor/rule_loader.rb:33:in \`load_rules'
 ./spec/datadog/appsec/processor/rule_loader_spec.rb:25:in \`block in <main>'
 ./spec/datadog/appsec/processor/rule_loader_spec.rb:69:in \`block in <main>'
 ...

  • Datadog::AppSec::Processor::RuleLoader#load_ruleset when ruleset is not parseable returns nil`` - rspec - Details

    Expand for error 
     Unexpected Datadog.logger.error():
 
 Failure/Error: raise "Unexpected Datadog.logger.error(): #{msg}"
 
 RuntimeError:
   Unexpected Datadog.logger.error(): 
 ./spec/support/log_helpers.rb:24:in \`block in included'
 ./lib/datadog/appsec/processor/rule_loader.rb:33:in \`load_rules'
 ./spec/datadog/appsec/processor/rule_loader_spec.rb:25:in \`block in <main>'
 ./spec/datadog/appsec/processor/rule_loader_spec.rb:100:in \`block in <main>'
 ...

New Flaky Tests (169)

  • ActiveJob with active_job instrumentation instruments active_job.enqueue_retry and active_job.retry_stopped - rspec - Last Failure

    Expand for error 
     
 RuntimeError:
 ./spec/support/log_helpers.rb:12:in \`block (2 levels) in included'
 ./lib/datadog/tracing/contrib/extensions.rb:97:in \`block in configure'
 ./lib/datadog/tracing/contrib/extensions.rb:78:in \`configure'
 ./spec/datadog/tracing/contrib/rails/rails_active_job_spec.rb:53:in \`block (3 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/rails/support/log_configuration.rb:7:in \`block (2 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/support/tracer_helpers.rb:96:in \`block (2 levels) in <module:TracerHelpers>'
 ./spec/spec_helper.rb:266:in \`block (2 levels) in <top (required)>'
 ./spec/spec_helper.rb:151:in \`block (2 levels) in <top (required)>'
 ...

  • ActiveJob with active_job instrumentation instruments discard - rspec - Last Failure

    Expand for error 
     
 RuntimeError:
 ./spec/support/log_helpers.rb:12:in \`block (2 levels) in included'
 ./lib/datadog/tracing/contrib/extensions.rb:97:in \`block in configure'
 ./lib/datadog/tracing/contrib/extensions.rb:78:in \`configure'
 ./spec/datadog/tracing/contrib/rails/rails_active_job_spec.rb:53:in \`block (3 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/rails/support/log_configuration.rb:7:in \`block (2 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/support/tracer_helpers.rb:96:in \`block (2 levels) in <module:TracerHelpers>'
 ./spec/spec_helper.rb:266:in \`block (2 levels) in <top (required)>'
 ./spec/spec_helper.rb:151:in \`block (2 levels) in <top (required)>'
 ...

  • ActiveJob with active_job instrumentation instruments enqueue - rspec - Last Failure

    Expand for error 
     
 RuntimeError:
 ./spec/support/log_helpers.rb:12:in \`block (2 levels) in included'
 ./lib/datadog/tracing/contrib/extensions.rb:97:in \`block in configure'
 ./lib/datadog/tracing/contrib/extensions.rb:78:in \`configure'
 ./spec/datadog/tracing/contrib/rails/rails_active_job_spec.rb:53:in \`block (3 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/rails/support/log_configuration.rb:7:in \`block (2 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/support/tracer_helpers.rb:96:in \`block (2 levels) in <module:TracerHelpers>'
 ./spec/spec_helper.rb:266:in \`block (2 levels) in <top (required)>'
 ./spec/spec_helper.rb:151:in \`block (2 levels) in <top (required)>'
 ...

  • ActiveJob with active_job instrumentation instruments enqueue_at under the "enqueue" span - rspec - Last Failure

    Expand for error 
     
 RuntimeError:
 ./spec/support/log_helpers.rb:12:in \`block (2 levels) in included'
 ./lib/datadog/tracing/contrib/extensions.rb:97:in \`block in configure'
 ./lib/datadog/tracing/contrib/extensions.rb:78:in \`configure'
 ./spec/datadog/tracing/contrib/rails/rails_active_job_spec.rb:53:in \`block (3 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/rails/support/log_configuration.rb:7:in \`block (2 levels) in <top (required)>'
 ./spec/datadog/tracing/contrib/support/tracer_helpers.rb:96:in \`block (2 levels) in <module:TracerHelpers>'
 ./spec/spec_helper.rb:266:in \`block (2 levels) in <top (required)>'
 ./spec/spec_helper.rb:151:in \`block (2 levels) in <top (required)>'
 ...

  • ActiveJob with active_job instrumentation instruments perform - rspec - Last Failure

    Expand for error 
     
 RuntimeError:
 ./spec/support/log_helpers.rb:12:in \`block (2 levels) in included'
 ./lib/datadog/tracing/contrib/extensions.rb:97:in \`block in configure'
 ./lib/datadog/tracing/contrib/extensions.rb:78:in \`configure'
 ./lib/datadog/tracing/contrib/rails/framework.rb:37:in \`setup'
 ./lib/datadog/tracing/contrib/rails/patcher.rb:84:in \`setup_tracer'
 ./lib/datadog/tracing/contrib/rails/patcher.rb:78:in \`block in after_initialize'
 ./lib/datadog/core/utils/only_once.rb:26:in \`block in run'
 ./lib/datadog/core/utils/only_once.rb:21:in \`synchronize'
 ...

@pr-commenter
Copy link

pr-commenter bot commented May 21, 2025 •
edited
Loading

Benchmarks

Benchmark execution time: 2025-05-21 14:19:44

Comparing candidate commit 389084c in PR branch tmp/marco-test-this with baseline commit 2deebe6 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 39 metrics, 6 unstable metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dev/testing Involves testing processes (e.g. RSpec)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@marcotc