Check for a valid trace for ALL tests

[plbossart]

We need to double-check that the trace is functional and that sof-logger reports at least the firmware details and a dai trigger.

If there is no trace then there's no point in checking results and debugging further.

cc:

• Catch firmware errors #1075
• boards: Intel cAVS: disable DTRACE sof#7955
• Catch firmware errors #1075
• lib.sh: add get_ldc_subdir() that follows /sys/module/snd_sof_pci/parameters/fw_path #1084

• many others, follow the links.

[marc-hb]

First part submitted in #373, please help review.

[marc-hb]

From @xiulipan in #373

it that behavior right? Do we need to fail the case if we could not get the logger? As we have standalone test case for logger.

[marc-hb]

Do we need to fail the case if we could not get the logger?

I think what you're asking is: "is the logger a low priority feature"? I read the description of this request as a pretty clear "no" but I'd like to hear more from others.

As we have standalone test case for logger.

It's risky to rely on a single test, if the test is broken then the feature can be broken and no one will notice. Even if the test is not broken, the logger could be broken only in some scenarios not tested by the logger test. I think the links above are some real-world examples of either.

A test is never meant to test only one thing, the more extras each test can test (in no extra time) and the better. Unexpected failures mean extra bugs are found which is great.

These questions sounds strange to me at a time when the logger seems to be working fine, I mean all tests in #373 pass right now. I mean it sounds very strange to worry about failures we don't even have right now (this reminds me of recent #372) It almost sounds like like a desire to find FEWER bugs / and have more "green failures"?

[xiulipan]

@marc-hb What I want to make clarify about is the criterion for our test case pass/fail.
Should we make aplay test fail if a debug feature is not working?

A test is never meant to test only one thing, the more extras each test can test (in no extra time) and the better.

I do not think we are adding test cases in this way. Each test case in this repo is focusing in a single feature. This will help us to narrow down the issue.

[marc-hb]

I do not think we are adding test cases in this way. Each test case in this repo is focusing in a single feature. This will help us to narrow down the issue.

This is :

• neither possible because sof-test are not unit tests. They all run from user space so each test involves many lines of code
• nor desirable unless you want to find fewer bugs - as it happened in the links above and elsewhere.

[marc-hb]

I mean all tests in #373 pass right now.

They didn't pass, they only appear to pass because I misused exit_failure=2 in #373 where 2 means "Not Applicable" and Not Applicable is green. I didn't know 2 was a special value.

I mean it sounds very strange to worry about [logger] failures we don't even have right now

Now we have one: pretty much all tests involving the logger fail to kill -INT it at the end, see new bug https://github.com/thesofproject/sof/issues/3433 [BUG] sof-logger resisted pkill -INT, using -KILL

[marc-hb]

multipipeline tests returning an empty trace: #380 (comment)

[marc-hb]

multiple-pipeline-capture.sh and multiple-pipeline-playback.sh have a consistently empty trace on about half the platforms: https://sof-ci.01.org/softestpr/PR383/build194/devicetest/

2020-09-18 05:13:41 UTC [REMOTE_INFO] nlines=1 /home/ubuntu/sof-test/logs/multiple-pipeline-playback/2020-09-18-05:13:29-16938/slogger.txt
2020-09-18 05:13:41 UTC [REMOTE_ERROR] Empty logger trace
2020-09-18 05:13:41 UTC [REMOTE_INFO] Test Result: FAIL!

@aiChaoSONG does this seem related to your #359 PR?

[aiChaoSONG]

@marc-hb Yes, they are related. without #359, these two case are actually testing nothing, and always pass. It is the sof-logger issue which makes the multiple-pipeline-playback/capture failed after your #373 patch