Console log: 'Microsoft.ML.GenAI.LLaMA.Tests' from job 49d64c01-a3ea-410e-8a50-b6f2c715bd07 (ubuntu.2204.amd64.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-mlnet-helix on a009YRY
running $HELIX_CORRELATION_PAYLOAD/scripts/bc7ec8154b214f4ab0e6e2d1b6804997/execute.sh in /datadisks/disk1/work/ABE90963/w/B0840942/e max 3600 seconds
Output:
[BEGIN EXECUTION]
+ [ -e /home/helixbot/.vsts-env/bin/activate ]
+ [ -z /home/helixbot/.vsts-env ]
+ echo /home/helixbot/.vsts-env
+ grep -qv ^/home/helixbot/.vsts-env
+ echo /home/helixbot/.vsts-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ grep -Eqv ^(^|:)/home/helixbot/.vsts-env/bin(:|$)
+ chown -R helixbot /home/helixbot/.vsts-env
+ chmod -R u+w /home/helixbot/.vsts-env
+ command -v python3
+ HELIX_PYTHONPATH=/home/helixbot/.vsts-env/bin/python3
+ export HELIX_PYTHONPATH
+ unset PYTHONPATH
+ /home/helixbot/.vsts-env/bin/python3 -m pip list --disable-pip-version-check
+ grep ^helix-scripts
helix-scripts 20250318.5
+ pipCommand=VIRTUAL_ENV=/home/helixbot/.vsts-env /home/helixbot/.vsts-env/bin/python3 -m pip uninstall --disable-pip-version-check --yes helix-scripts
+ command -v sudo
+ [ -z /usr/bin/sudo ]
+ [ -n /home/helixbot/.vsts-env ]
+ eval VIRTUAL_ENV=/home/helixbot/.vsts-env /home/helixbot/.vsts-env/bin/python3 -m pip uninstall --disable-pip-version-check --yes helix-scripts
+ VIRTUAL_ENV=/home/helixbot/.vsts-env /home/helixbot/.vsts-env/bin/python3 -m pip uninstall --disable-pip-version-check --yes helix-scripts
Found existing installation: helix-scripts 20250318.5
Uninstalling helix-scripts-20250318.5:
Successfully uninstalled helix-scripts-20250318.5
+ pipCommand=VIRTUAL_ENV=/home/helixbot/.vsts-env /home/helixbot/.vsts-env/bin/python3 -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
+ command -v sudo
+ [ -z /usr/bin/sudo ]
+ [ -n /home/helixbot/.vsts-env ]
+ eval VIRTUAL_ENV=/home/helixbot/.vsts-env /home/helixbot/.vsts-env/bin/python3 -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
+ VIRTUAL_ENV=/home/helixbot/.vsts-env /home/helixbot/.vsts-env/bin/python3 -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
Ignoring azure-core: markers 'python_version >= "3.12"' don't match your environment
Ignoring azure-identity: markers 'python_version >= "3.12"' don't match your environment
Ignoring cffi: markers 'python_version < "3.8"' don't match your environment
Ignoring importlib_metadata: markers 'python_version < "3.8"' don't match your environment
Requirement already satisfied: asn1crypto==0.24.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 1)) (0.24.0)
Requirement already satisfied: azure-core~=1.24 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 3)) (1.32.0)
Requirement already satisfied: azure-devops==5.0.0b9 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 4)) (5.0.0b9)
Requirement already satisfied: azure-identity~=1.10 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 6)) (1.21.0)
Requirement already satisfied: azure-storage-blob==12.5.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 7)) (12.5.0)
Requirement already satisfied: certifi>=2023.7.22 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 8)) (2025.1.31)
Requirement already satisfied: cffi>=1.16.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 10)) (1.17.1)
Requirement already satisfied: cryptography>=39.0.1 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 11)) (44.0.2)
Requirement already satisfied: docker-pycreds==0.4.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 12)) (0.4.0)
Requirement already satisfied: docker==3.7.2 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 13)) (3.7.2)
Requirement already satisfied: idna==2.8 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 14)) (2.8)
Requirement already satisfied: isodate==0.6.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 16)) (0.6.0)
Requirement already satisfied: msrest==0.6.10 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 17)) (0.6.10)
Requirement already satisfied: ndg-httpsclient==0.5.1 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 18)) (0.5.1)
Requirement already satisfied: oauthlib==3.1.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 19)) (3.1.0)
Requirement already satisfied: packaging in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (24.2)
Requirement already satisfied: psutil>=5.9.3 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 21)) (7.0.0)
Requirement already satisfied: pyasn1==0.4.5 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 22)) (0.4.5)
Requirement already satisfied: pycparser==2.19 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 23)) (2.19)
Requirement already satisfied: pyopenssl>=23.0.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 24)) (25.0.0)
Requirement already satisfied: pyyaml>=3.11 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 25)) (6.0.2)
Requirement already satisfied: requests-oauthlib==1.2.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 26)) (1.2.0)
Requirement already satisfied: requests<=2.28.1,>=2.21.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 27)) (2.28.1)
Requirement already satisfied: setuptools in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 28)) (59.6.0)
Requirement already satisfied: six>=1.11.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 29)) (1.17.0)
Requirement already satisfied: urllib3<1.27,>=1.24.3 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 30)) (1.26.20)
Requirement already satisfied: websocket-client==0.56.0 in ./.vsts-env/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 31)) (0.56.0)
Requirement already satisfied: typing-extensions>=4.6.0 in ./.vsts-env/lib/python3.10/site-packages (from azure-core~=1.24->-r /root/helix/scripts/runtime_python_requirements.txt (line 3)) (4.12.2)
Requirement already satisfied: msal>=1.30.0 in ./.vsts-env/lib/python3.10/site-packages (from azure-identity~=1.10->-r /root/helix/scripts/runtime_python_requirements.txt (line 6)) (1.32.0)
Requirement already satisfied: msal-extensions>=1.2.0 in ./.vsts-env/lib/python3.10/site-packages (from azure-identity~=1.10->-r /root/helix/scripts/runtime_python_requirements.txt (line 6)) (1.3.1)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.vsts-env/lib/python3.10/site-packages (from requests<=2.28.1,>=2.21.0->-r /root/helix/scripts/runtime_python_requirements.txt (line 27)) (2.1.1)
Requirement already satisfied: PyJWT[crypto]<3,>=1.0.0 in ./.vsts-env/lib/python3.10/site-packages (from msal>=1.30.0->azure-identity~=1.10->-r /root/helix/scripts/runtime_python_requirements.txt (line 6)) (2.10.1)
+ PYTHONPATH=/root/helix/scripts
+ export PYTHONPATH
+ cd /root/helix/work/workitem/e
+ mkdir -p /home/helixbot/dotnetbuild/dumps/
+ /root/helix/work/correlation/scripts/bc7ec8154b214f4ab0e6e2d1b6804997/execute.sh
+ export ML_TEST_DATADIR=/root/helix/work/correlation
+ export MICROSOFTML_RESOURCE_PATH=/root/helix/work/workitem/e
+ sudo chmod -R 777 /root/helix/work/workitem/e
+ sudo chown -R /root/helix/work/workitem/e
chown: missing operand after '/root/helix/work/workitem/e'
Try 'chown --help' for more information.
+ export PATH=/root/helix/work/correlation/dotnet-cli:/home/helixbot/.vsts-env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ ./runTests.sh
----- start Wed Mar 19 22:45:17 UTC 2025 =============== To repro directly: =====================================================
pushd .
dotnet exec --roll-forward Major --runtimeconfig Microsoft.ML.GenAI.LLaMA.Tests.runtimeconfig.json --depsfile Microsoft.ML.GenAI.LLaMA.Tests.deps.json /root/helix/work/correlation/xunit-runner/tools/netcoreapp2.0/xunit.console.dll Microsoft.ML.GenAI.LLaMA.Tests.dll -notrait Category=SkipInCI -xml testResults.xml
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
xUnit.net Console Runner v2.4.2+f110e5bee5 (64-bit .NET 8.0.1)
[37m Discovering: Microsoft.ML.GenAI.LLaMA.Tests
[m[37m Discovered: Microsoft.ML.GenAI.LLaMA.Tests
[m[37m Starting: Microsoft.ML.GenAI.LLaMA.Tests
./runTests.sh: line 25: 36 Killed dotnet exec --roll-forward Major --runtimeconfig Microsoft.ML.GenAI.LLaMA.Tests.runtimeconfig.json --depsfile Microsoft.ML.GenAI.LLaMA.Tests.deps.json $HELIX_CORRELATION_PAYLOAD/xunit-runner/tools/netcoreapp2.0/xunit.console.dll Microsoft.ML.GenAI.LLaMA.Tests.dll -notrait Category=SkipInCI -xml testResults.xml
[m/root/helix/work/workitem/e
----- end Wed Mar 19 22:45:26 UTC 2025 ----- exit code 137 ----------------------------------------------------------
tar: ./TestOutput: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
+ export _commandExitCode=137
+ /home/helixbot/.vsts-env/bin/python3 /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 26322124 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Imt6UFh2cVJPMEN1UzRqU296REc4d21EM1RmcyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBQaXBlbGluZUNhY2hlLlJlYWRXcml0ZVJvb3RBY2Nlc3MgUmVhZEFuZFB1Ymxpc2hUZXN0OmNiYjE4MjYxLWM0OGYtNGFiYi04NjUxLThjZGNiNTQ3NDY0OSBSZWFkQW5kVXBkYXRlQnVpbGRCeVVyaTpjYmIxODI2MS1jNDhmLTRhYmItODY1MS04Y2RjYjU0NzQ2NDkvZG90bmV0L21hY2hpbmVsZWFybmluZy8xNjc6QnVpbGQvQnVpbGQvOTg2NjU2IiwiYXVpIjoiYjBkMWYyNDUtNmZkNS00ZjZiLTk3YTctODY0YWIzZTg2OTQ5Iiwic2lkIjoiODU0YTgyOGYtNDNjYi00OWEyLTg5MjMtM2VkYjBlOTgzY2JjIiwiQnVpbGRJZCI6ImNiYjE4MjYxLWM0OGYtNGFiYi04NjUxLThjZGNiNTQ3NDY0OTs5ODY2NTYiLCJEZWZJZCI6IjE2NyIsImpvYnJlZiI6IjQ2YjczMDU2LWMwMDEtNGFmNy05MGEwLTRhZTM2MjllZmIzMzo2NjBlOGViNS1kNWVlLTU2NTMtNzc4ZC0zNDkwYjhmMDY5OTMiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvOTg2NjU2Iiwib3JjaGlkIjoiNDZiNzMwNTYtYzAwMS00YWY3LTkwYTAtNGFlMzYyOWVmYjMzLnVidW50dV94NjQucmVsZWFzZV9idWlsZCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNzQyNDIyNzEzLCJleHAiOjE3NDI0MzE0MTN9.yV5DDdtldPf-hiHpmOAzRgPX4tvREi6jG9HWRWyQdKz3RMN4ZYi07ZYozvBNn3J2-BIMIfu2dzcneqyWq4F44by-nB_qtdx5eULz_rcBCww2BJhMtgvJ9UUfDORdzNFQ12Gvs0x4gq_OTw5COyWS5UzaTjpFf2jdCSfO6igUdRsY-8wv9iz3Sq0eM7Lzi43NvGhInb6Dm-UfyGJAjY1QZ2BdWk5LMc8udgpiZsr2YYcZ-usA2H5NtJ5IRR5qJP-iJ9gTGR_AhWGs9J2sVsQipFUtCT0B5l7WHSCaojMmjWi9fCnihO5SAX75fKKyDQ1RmgnOaFsLGmeM36XsoQpWaw
2025-03-19T22:45:29.712Z INFO run.py managed_identity(145) __init__ ManagedIdentityCredential will use IMDS with client_id: 3423231b-6b9e-4177-8817-76276b647404
2025-03-19T22:45:29.715Z INFO run.py azure_utils(24) get_credential_and_access_token VMSSManagedIdentity Auth type will be used
2025-03-19T22:45:29.769Z INFO run.py _universal(509) on_request Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED&client_id=REDACTED'
Request method: 'GET'
Request headers:
'Metadata': 'REDACTED'
'User-Agent': 'azsdk-python-identity/1.21.0 Python/3.10.12 (Linux-6.8.0-1021-azure-x86_64-with-glibc2.35)'
No body was attached to the request
2025-03-19T22:45:29.782Z INFO run.py _universal(545) on_response Response status: 200
Response headers:
'Content-Type': 'application/json; charset=utf-8'
'Server': 'IMDS/150.870.65.1544'
'x-ms-request-id': 'd4b6bf95-d4df-4472-a4ae-fb16ac908ddd'
'Date': 'Wed, 19 Mar 2025 22:45:29 GMT'
'Content-Length': '2051'
2025-03-19T22:45:29.785Z INFO run.py msal_managed_identity_client(165) _get_token_base ImdsCredential.get_token succeeded
2025-03-19T22:45:29.785Z INFO run.py decorators(24) wrapper ManagedIdentityCredential.get_token succeeded
2025-03-19T22:45:29.785Z INFO run.py azure_utils(53) get_credential_and_access_token Credentials are valid
2025-03-19T22:45:29.808Z INFO run.py _universal(509) on_request Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED&client_id=REDACTED'
Request method: 'GET'
Request headers:
'Metadata': 'REDACTED'
'User-Agent': 'azsdk-python-identity/1.21.0 Python/3.10.12 (Linux-6.8.0-1021-azure-x86_64-with-glibc2.35)'
No body was attached to the request
2025-03-19T22:45:29.814Z INFO run.py _universal(545) on_response Response status: 200
Response headers:
'Content-Type': 'application/json; charset=utf-8'
'Server': 'IMDS/150.870.65.1544'
'x-ms-request-id': '989622f2-bc48-490e-a03f-bc6a326dea35'
'Date': 'Wed, 19 Mar 2025 22:45:29 GMT'
'Content-Length': '2051'
2025-03-19T22:45:29.815Z INFO run.py msal_managed_identity_client(165) _get_token_base ImdsCredential.get_token_info succeeded
2025-03-19T22:45:29.815Z INFO run.py decorators(24) wrapper ManagedIdentityCredential.get_token_info succeeded
2025-03-19T22:45:30.065Z INFO run.py run(48) main Beginning reading of test results.
2025-03-19T22:45:30.066Z INFO run.py __init__(42) read_results Searching '/root/helix/work/workitem/e' for test results files
2025-03-19T22:45:30.149Z INFO run.py __init__(42) read_results Searching '/root/helix/work/workitem/uploads' for test results files
2025-03-19T22:45:30.150Z WARNING run.py __init__(55) read_results No results file found in any of the following formats: xunit, junit, trx
2025-03-19T22:45:30.150Z INFO run.py packing_test_reporter(30) report_results Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2025-03-19T22:45:30.150Z INFO run.py packing_test_reporter(33) report_results Packed 1798 bytes
+ exit 137
+ _commandExitCode=137
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 137
[END EXECUTION]
Exit Code:137