Run functional tests with -races

[tleyden]

Retest #1260 and look for other issues.

[tleyden]

After running this functional test:

python -m pytest --capture=no functional_tests/test_users_channels.py::test_single_user_single_channel

I saw some gocb data race errors:

https://issues.couchbase.com/browse/GOCBC-82

[tleyden]

Note to self: can run provision.py script with -race flag

[tleyden]

In progress

From @sethrosetter:

http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests-RACES/

There is your sandbox. Steps to get this working:

1. Configure so that you can pass in a build flag to provision_cluster.py. Perhaps a branch and target that in the job.
2. Configure the above job to build from source with the races flag instead of installing the package (current behavior).

[tleyden]

Waiting for job to start;

http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests-RACES/ (job #1)

[tleyden]

Change needed on testcluster stuff: couchbaselabs/mobile-testkit#242

[tleyden]

Job in progress: http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests-RACES/5/console

[tleyden]

The test above had many failures, however I grepped all of the sync gateway logs of the failed cases and did not find the DATA RACE string.

[tleyden]

http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests-RACES/7/ will have the explicit checking of the SG logs for panics / races on any test failures.

[tleyden]

In progress:

http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests-RACES/16/console

this is checking every log zipfile for "panic" or "DATA RACE", even for passing tests.

[tleyden]

Detected races:

2016-01-25 19:09:51,016 - ERROR - fetch_sync_gateway_logs - Error pulling logs
2016-01-25 19:09:56,457 - INFO - detected_pattern - Detected pattern DATA RACE: sg3/sync_gateway_error.log:WARNING: DATA RACE
sg3/sync_gateway_error.log:WARNING: DATA RACE
sg3/sync_gateway_error.log:WARNING: DATA RACE
sg3/sync_gateway_error.log:WARNING: DATA RACE

2016-01-25 19:09:56,460 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_continuous.py-test_continuous_changes_parametrized[CC-6]-2016-01-25-19-09-51-sglogs.zip

[tleyden]

The job finished and found more race issues:

http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests-RACES/16/consoleFull

2016-01-25 21:33:27,848 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_db_online_offline.py-test_db_delayed_online[CC-1]-2016-01-25-21-33-27-sglogs.zip
2016-01-25 22:12:45,656 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_multiple_dbs.py-test_multiple_db_unique_data_bucket_unique_index_bucket[CC-2]-2016-01-25-22-12-44-sglogs.zip
2016-01-25 22:23:07,529 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_multiple_dbs.py-test_multiple_db_single_data_bucket_single_index_bucket[CC-2]-2016-01-25-22-23-05-sglogs.zip
2016-01-25 23:17:41,789 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_seq.py-test_seq[CC-2]-2016-01-25-23-17-40-sglogs.zip

I will dig into the logs and file tickets for all the ones that look to be unique.

[tleyden]

2016-01-25 21:33:27,848 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_db_online_offline.py-test_db_delayed_online[CC-1]-2016-01-25-21-33-27-sglogs.zip

Filed: couchbase/go-couchbase#72

[tleyden]

2016-01-25 22:12:45,656 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_multiple_dbs.py-test_multiple_db_unique_data_bucket_unique_index_bucket[CC-2]-2016-01-25-22-12-44-sglogs.zip
2016-01-25 22:23:07,529 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_multiple_dbs.py-test_multiple_db_single_data_bucket_single_index_bucket[CC-2]-2016-01-25-22-23-05-sglogs.zip
2016-01-25 23:17:41,789 - ERROR - fetch_logs - Detected data races in logs: /tmp/functional_tests-test_seq.py-test_seq[CC-2]-2016-01-25-23-17-40-sglogs.zip

#1521