Console log: 'Microsoft.ML.TensorFlow.Tests' from job 40678905-aad8-452b-9807-6d34e4854dfe workitem 888888c6-3578-465d-a105-0d8688ea5cdb (windows.10.amd64.open) executed on machine a00EBM3 running Windows-2016Server-10.0.14393-SP0 C:\h\w\9D2C08F9\w\9AA10905\e>set ML_TEST_DATADIR=C:\h\w\9D2C08F9\p C:\h\w\9D2C08F9\w\9AA10905\e>set MICROSOFTML_RESOURCE_PATH=C:\h\w\9D2C08F9\w\9AA10905\e C:\h\w\9D2C08F9\w\9AA10905\e>set PATH=C:\h\w\9D2C08F9\p\dotnet-cli;C:\python3\Scripts\;C:\python3\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Debuggers\x64;C:\Program Files\Microsoft SQL Server\160\Tools\Binn\;C:\Users\runner\AppData\Local\Microsoft\WindowsApps C:\h\w\9D2C08F9\w\9AA10905\e>call .\runTests.cmd ----- start Tue 04/14/2026 6:20:57.45 =============== To repro directly: ===================================================== pushd C:\h\w\9D2C08F9\w\9AA10905\e\ C:\h\w\9D2C08F9\p/xunit-runner/tools/net48/xunit.console.exe Microsoft.ML.TensorFlow.Tests.dll -notrait Category=SkipInCI -xml testResults.xml popd =========================================================================================================== C:\h\w\9D2C08F9\w\9AA10905\e>C:\h\w\9D2C08F9\p/xunit-runner/tools/net48/xunit.console.exe Microsoft.ML.TensorFlow.Tests.dll -notrait Category=SkipInCI -xml testResults.xml xUnit.net Console Runner v2.9.3+9712244020 (64-bit .NET Framework 4.8, runtime: 4.0.30319.42000) Discovering: Microsoft.ML.TensorFlow.Tests 2026-04-14 06:21:02.097768: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.167749: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.257064: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.318277: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.375915: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.433714: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.501534: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.583394: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.659870: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.734760: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.815273: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.900737: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:02.967132: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.025524: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.083142: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.142582: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.203132: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.262988: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.325661: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.399927: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.460295: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.519033: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.593974: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.676762: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.753969: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.832486: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.912908: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:03.977508: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:04.051964: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:04.128703: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:04.200812: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:04.260804: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:04.332078: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:04.396847: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Discovered: Microsoft.ML.TensorFlow.Tests Starting: Microsoft.ML.TensorFlow.Tests Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformInputShapeTest 2026-04-14 06:21:06.702960: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2026-04-14 06:21:06.727320: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: model_shape_test 2026-04-14 06:21:06.728340: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:21:06.728702: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: model_shape_test 2026-04-14 06:21:06.729613: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: SSE SSE2 SSE3 SSE4.1 SSE4.2 AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2026-04-14 06:21:06.731765: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled 2026-04-14 06:21:06.733335: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 5472 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformInputShapeTest with memory usage 620,933,120.00 and max memory usage 621,576,192.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationDefault Saver not created because there are no variables in the graph to restore Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\znjbvdv2.wnq\custom_retrained_model_based_on_resnet_v2_50_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationDefault with memory usage 983,678,976.00 and max memory usage 2,633,158,656.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationWithExponentialLRScheduling Saver not created because there are no variables in the graph to restore Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 10 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 11 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 12 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 13 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 14 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 15 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 16 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 17 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 18 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 19 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 20 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 21 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 22 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 23 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 24 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 25 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 26 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 27 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 28 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 29 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 30 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 31 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 32 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 33 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 34 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 35 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 36 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 37 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 38 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 39 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 40 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 41 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 0, Accuracy: 0.52, Cross-Entropy: 1.320662, Learning Rate: 0.01 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 0, Accuracy: 0.7777778, Cross-Entropy: 0.5583564 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 1, Accuracy: 0.82, Cross-Entropy: 0.6798154, Learning Rate: 0.0094 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 1, Accuracy: 0.8888889, Cross-Entropy: 0.3473784 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 2, Accuracy: 1, Cross-Entropy: 0.344289, Learning Rate: 0.0094 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 2, Accuracy: 1, Cross-Entropy: 0.2680032 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.2493653, Learning Rate: 0.008836 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.2291326 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.1942611, Learning Rate: 0.008836 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.2045116 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.1609842, Learning Rate: 0.008305839 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.1881682 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.1371654, Learning Rate: 0.008305839 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.1756946 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.1205663, Learning Rate: 0.007807489 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.1663124 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.107414, Learning Rate: 0.007807489 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.1585344 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.09752752, Learning Rate: 0.00733904 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.152314 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.0892198, Learning Rate: 0.00733904 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.1469233 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.08267941, Learning Rate: 0.006898697 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.1424508 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.07696954, Learning Rate: 0.006898697 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.1384668 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.07233287, Learning Rate: 0.006484775 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.1350783 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.0681753, Learning Rate: 0.006484775 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.132003 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.06472355, Learning Rate: 0.006095689 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.1293397 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.06156667, Learning Rate: 0.006095689 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.1268897 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.05890199, Learning Rate: 0.005729948 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.1247384 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.05642775, Learning Rate: 0.005729948 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.1227391 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.05431236, Learning Rate: 0.005386151 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.1209643 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.05232437, Learning Rate: 0.005386151 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.1193016 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.05060732, Learning Rate: 0.005062982 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.1178125 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 22, Accuracy: 1, Cross-Entropy: 0.04897796, Learning Rate: 0.005062982 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 22, Accuracy: 1, Cross-Entropy: 0.1164085 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 23, Accuracy: 1, Cross-Entropy: 0.04755897, Learning Rate: 0.004759203 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 23, Accuracy: 1, Cross-Entropy: 0.115142 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 24, Accuracy: 1, Cross-Entropy: 0.04620159, Learning Rate: 0.004759203 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 24, Accuracy: 1, Cross-Entropy: 0.1139415 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 25, Accuracy: 1, Cross-Entropy: 0.04501133, Learning Rate: 0.004473651 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 25, Accuracy: 1, Cross-Entropy: 0.1128519 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 26, Accuracy: 1, Cross-Entropy: 0.04386511, Learning Rate: 0.004473651 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 26, Accuracy: 1, Cross-Entropy: 0.1118146 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 27, Accuracy: 1, Cross-Entropy: 0.04285423, Learning Rate: 0.004205232 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 27, Accuracy: 1, Cross-Entropy: 0.1108681 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 28, Accuracy: 1, Cross-Entropy: 0.04187518, Learning Rate: 0.004205232 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 28, Accuracy: 1, Cross-Entropy: 0.1099639 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 29, Accuracy: 1, Cross-Entropy: 0.04100755, Learning Rate: 0.003952918 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 29, Accuracy: 1, Cross-Entropy: 0.1091351 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 30, Accuracy: 1, Cross-Entropy: 0.04016306, Learning Rate: 0.003952918 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 30, Accuracy: 1, Cross-Entropy: 0.1083407 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 31, Accuracy: 1, Cross-Entropy: 0.03941157, Learning Rate: 0.003715743 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 31, Accuracy: 1, Cross-Entropy: 0.1076099 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 32, Accuracy: 1, Cross-Entropy: 0.03867701, Learning Rate: 0.003715743 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 32, Accuracy: 1, Cross-Entropy: 0.1069076 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 33, Accuracy: 1, Cross-Entropy: 0.038021, Learning Rate: 0.003492798 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 33, Accuracy: 1, Cross-Entropy: 0.1062592 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 34, Accuracy: 1, Cross-Entropy: 0.03737739, Learning Rate: 0.003492798 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 34, Accuracy: 1, Cross-Entropy: 0.1056345 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 35, Accuracy: 1, Cross-Entropy: 0.03680073, Learning Rate: 0.00328323 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 35, Accuracy: 1, Cross-Entropy: 0.1050562 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 36, Accuracy: 1, Cross-Entropy: 0.03623318, Learning Rate: 0.00328323 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 36, Accuracy: 1, Cross-Entropy: 0.1044979 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 37, Accuracy: 1, Cross-Entropy: 0.03572326, Learning Rate: 0.003086236 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 37, Accuracy: 1, Cross-Entropy: 0.1039796 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 38, Accuracy: 1, Cross-Entropy: 0.03521999, Learning Rate: 0.003086236 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 38, Accuracy: 1, Cross-Entropy: 0.1034784 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 39, Accuracy: 1, Cross-Entropy: 0.03476661, Learning Rate: 0.002901062 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 39, Accuracy: 1, Cross-Entropy: 0.1030121 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 40, Accuracy: 1, Cross-Entropy: 0.03431803, Learning Rate: 0.002901062 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 40, Accuracy: 1, Cross-Entropy: 0.1025604 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 41, Accuracy: 1, Cross-Entropy: 0.03391311, Learning Rate: 0.002726999 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 41, Accuracy: 1, Cross-Entropy: 0.1021392 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 42, Accuracy: 1, Cross-Entropy: 0.0335115, Learning Rate: 0.002726999 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 42, Accuracy: 1, Cross-Entropy: 0.1017306 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 43, Accuracy: 1, Cross-Entropy: 0.0331483, Learning Rate: 0.002563379 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 43, Accuracy: 1, Cross-Entropy: 0.101349 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 44, Accuracy: 1, Cross-Entropy: 0.0327873, Learning Rate: 0.002563379 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 44, Accuracy: 1, Cross-Entropy: 0.1009783 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 45, Accuracy: 1, Cross-Entropy: 0.03246031, Learning Rate: 0.002409576 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 45, Accuracy: 1, Cross-Entropy: 0.1006315 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 46, Accuracy: 1, Cross-Entropy: 0.03213467, Learning Rate: 0.002409576 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 46, Accuracy: 1, Cross-Entropy: 0.1002942 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 47, Accuracy: 1, Cross-Entropy: 0.03183923, Learning Rate: 0.002265001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 47, Accuracy: 1, Cross-Entropy: 0.09997807 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 48, Accuracy: 1, Cross-Entropy: 0.03154451, Learning Rate: 0.002265001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 48, Accuracy: 1, Cross-Entropy: 0.09967039 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 49, Accuracy: 1, Cross-Entropy: 0.03127673, Learning Rate: 0.002129101 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 49, Accuracy: 1, Cross-Entropy: 0.09938172 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_0\custom_retrained_model_based_on_resnet_v2_101_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationWithExponentialLRScheduling with memory usage 1,509,138,432.00 and max memory usage 5,152,505,856.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifar Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifar with memory usage 1,523,322,880.00 and max memory usage 5,152,505,856.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransforCifarEndToEndTest2 Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransforCifarEndToEndTest2 with memory usage 1,702,891,520.00 and max memory usage 5,152,505,856.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationBadImages Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformObjectDetectionTest [SKIP] Model files are not available yet Saver not created because there are no variables in the graph to restore 2026-04-14 06:24:47.783377: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Trying to decode BMP format using a wrong op. Use `decode_bmp` or `decode_image` instead. Op used: DecodeJpeg [[{{node DecodeJpeg}}]] Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\h0xwbmak.ryh\custom_retrained_model_based_on_resnet_v2_101_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationBadImages with memory usage 1,548,857,344.00 and max memory usage 5,152,505,856.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTConvTest Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTConvTest with memory usage 1,551,790,080.00 and max memory usage 5,152,505,856.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorflowPlaceholderShapeInferenceTest Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorflowPlaceholderShapeInferenceTest with memory usage 1,552,015,360.00 and max memory usage 5,152,505,856.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification Saver not created because there are no variables in the graph to restore Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 0, Accuracy: 0.7777778, Cross-Entropy: 0.567695 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 1, Accuracy: 0.8888889, Cross-Entropy: 0.3518418 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 2, Accuracy: 0.8888889, Cross-Entropy: 0.2713999 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.2317664 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.2066662 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.1899866 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.177275 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.1677051 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.1597838 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.1534429 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.1479558 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.1433986 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.1393448 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.1358934 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.1327651 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.130053 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.1275612 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.1253709 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.1233377 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.121531 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.1198404 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.1183247 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 22, Accuracy: 1, Cross-Entropy: 0.1168973 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 23, Accuracy: 1, Cross-Entropy: 0.1156081 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_0\custom_retrained_model_based_on_resnet_v2_101_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification with memory usage 1,573,560,320.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification Saver not created because there are no variables in the graph to restore Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 10 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 11 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 12 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 13 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 14 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 15 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 16 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 17 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 18 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 19 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 20 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 21 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 22 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 23 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 24 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 25 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 26 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 27 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 28 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 29 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 30 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 31 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 32 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 33 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 34 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 35 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 36 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 37 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 38 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 39 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 40 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 41 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 0, Accuracy: 0.7777778, Cross-Entropy: 0.5469794 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 1, Accuracy: 1, Cross-Entropy: 0.3888435 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 2, Accuracy: 1, Cross-Entropy: 0.3283676 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.2915714 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.267033 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.2487675 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.2349719 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.2237851 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.2147985 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.2071465 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.2007647 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.1951554 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.1903582 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.1860472 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.1822938 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.1788654 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.1758402 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.1730426 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.1705486 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.1682196 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.1661264 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.1641564 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_2\custom_retrained_model_based_on_mobilenet_v2.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification with memory usage 1,799,618,560.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification Saver not created because there are no variables in the graph to restore Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 10 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 11 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 12 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 13 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 14 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 15 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 16 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 17 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 18 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 19 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 20 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 21 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 22 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 23 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 24 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 25 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 26 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 27 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 28 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 29 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 30 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 31 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 32 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 33 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 34 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 35 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 36 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 37 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 38 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 39 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 40 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 41 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 0, Accuracy: 0.7777778, Cross-Entropy: 0.6462076 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 1, Accuracy: 0.8888889, Cross-Entropy: 0.4067255 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 2, Accuracy: 0.8888889, Cross-Entropy: 0.3180119 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 3, Accuracy: 0.8888889, Cross-Entropy: 0.2775244 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 4, Accuracy: 0.8888889, Cross-Entropy: 0.2516586 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.2346616 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.2214821 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.2115215 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.2031725 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.1964333 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.1905691 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.1856461 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.1812683 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.1774932 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.1740878 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.1710926 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.1683634 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.1659258 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.1636877 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.1616643 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.1597955 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.1580891 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 22, Accuracy: 1, Cross-Entropy: 0.1565056 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 23, Accuracy: 1, Cross-Entropy: 0.1550477 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 24, Accuracy: 1, Cross-Entropy: 0.1536897 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 25, Accuracy: 1, Cross-Entropy: 0.1524307 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_3\custom_retrained_model_based_on_resnet_v2_50_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification with memory usage 1,604,763,648.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification Saver not created because there are no variables in the graph to restore Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Validation, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 1 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 2 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 3 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 4 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 5 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 6 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 7 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 8 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 9 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 10 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 11 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 12 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 13 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 14 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 15 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 16 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 17 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 18 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 19 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 20 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 21 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 22 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 23 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 24 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 25 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 26 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 27 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 28 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 29 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 30 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 31 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 32 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 33 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 34 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 35 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 36 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 37 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 38 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 39 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 40 Phase: Bottleneck Computation, Dataset used: Train, Image Index: 41 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 0, Accuracy: 0.7777778, Cross-Entropy: 0.6830682 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 1, Accuracy: 0.8888889, Cross-Entropy: 0.5347364 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 2, Accuracy: 0.8888889, Cross-Entropy: 0.4585926 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.4145823 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.3852021 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.3642488 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.3481447 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.3354491 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.3250019 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.3163127 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.3088751 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.3024808 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.2968661 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.2919295 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.2875169 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.283574 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.2800032 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.2767733 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.273819 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.2711212 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.2686339 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.2663454 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 22, Accuracy: 1, Cross-Entropy: 0.264222 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 23, Accuracy: 1, Cross-Entropy: 0.262256 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_1\custom_retrained_model_based_on_inception_v3.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification with memory usage 1,660,768,256.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowInputsOutputsSchemaTest Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowInputsOutputsSchemaTest with memory usage 1,687,105,536.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMatrixMultiplicationTest Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMatrixMultiplicationTest with memory usage 1,687,502,848.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTLRTrainingTest 2026-04-14 06:29:56.819579: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: mnist_lr_model 2026-04-14 06:29:56.822666: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:29:56.822885: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: mnist_lr_model 2026-04-14 06:29:56.824986: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:29:56.834297: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 14720 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTLRTrainingTest with memory usage 1,679,462,400.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTConvTrainingTest 2026-04-14 06:29:57.244006: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: mnist_conv_model 2026-04-14 06:29:57.246361: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:29:57.246666: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: mnist_conv_model 2026-04-14 06:29:57.252351: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:29:57.281193: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 37180 microseconds. 2026-04-14 06:29:57.789936: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: mnist_conv_model 2026-04-14 06:29:57.791640: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:29:57.791950: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: mnist_conv_model 2026-04-14 06:29:57.797883: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:29:57.830419: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 40477 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTConvTrainingTest with memory usage 1,684,951,040.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowSentimentClassificationTest Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformInceptionTest [SKIP] Model files are not available yet 2026-04-14 06:29:58.799586: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: sentiment_model 2026-04-14 06:29:58.803182: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:29:58.803402: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: sentiment_model 2026-04-14 06:29:58.817555: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:29:58.879041: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 79446 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowSentimentClassificationTest with memory usage 1,686,839,296.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifarSavedModel 2026-04-14 06:29:58.992879: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:29:58.994937: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:29:58.995160: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:29:59.000542: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:29:59.022926: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 30031 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifarSavedModel with memory usage 1,690,832,896.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformInputOutputTypesTest 2026-04-14 06:29:59.077721: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: model_types_test 2026-04-14 06:29:59.079130: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:29:59.079351: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: model_types_test 2026-04-14 06:29:59.080091: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 2371 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformInputOutputTypesTest with memory usage 1,691,271,168.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationEarlyStopping Saver not created because there are no variables in the graph to restore Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 0, Accuracy: 0.9, Cross-Entropy: 0.2944123 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 1, Accuracy: 1, Cross-Entropy: 0.1933667 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 2, Accuracy: 1, Cross-Entropy: 0.1599459 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.1424622 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.1307466 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.1230605 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.1166925 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.1119736 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.1078327 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.104587 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.1016531 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.09926818 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.09707086 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.09523778 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.09352598 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.09206997 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.0906966 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.08951084 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.08838367 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.08739893 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.08645697 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.08562611 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_0\custom_retrained_model_based_on_resnet_v2_101_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationEarlyStopping with memory usage 1,778,397,184.00 and max memory usage 5,157,285,888.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationEarlyStopping Saver not created because there are no variables in the graph to restore Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 0, Accuracy: 0.9, Cross-Entropy: 0.2911735 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 1, Accuracy: 1, Cross-Entropy: 0.190487 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 2, Accuracy: 1, Cross-Entropy: 0.1576689 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 3, Accuracy: 1, Cross-Entropy: 0.1404345 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 4, Accuracy: 1, Cross-Entropy: 0.1289111 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 5, Accuracy: 1, Cross-Entropy: 0.1213866 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 6, Accuracy: 1, Cross-Entropy: 0.1151548 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 7, Accuracy: 1, Cross-Entropy: 0.1105491 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 8, Accuracy: 1, Cross-Entropy: 0.1065073 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 9, Accuracy: 1, Cross-Entropy: 0.1033454 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 10, Accuracy: 1, Cross-Entropy: 0.1004865 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 11, Accuracy: 1, Cross-Entropy: 0.09816605 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 12, Accuracy: 1, Cross-Entropy: 0.09602748 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 13, Accuracy: 1, Cross-Entropy: 0.09424572 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 14, Accuracy: 1, Cross-Entropy: 0.09258126 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 15, Accuracy: 1, Cross-Entropy: 0.09116708 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 16, Accuracy: 1, Cross-Entropy: 0.08983272 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 17, Accuracy: 1, Cross-Entropy: 0.08868177 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 18, Accuracy: 1, Cross-Entropy: 0.08758725 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 19, Accuracy: 1, Cross-Entropy: 0.08663183 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 20, Accuracy: 1, Cross-Entropy: 0.08571768 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 21, Accuracy: 1, Cross-Entropy: 0.08491199 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 22, Accuracy: 1, Cross-Entropy: 0.08413713 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 23, Accuracy: 1, Cross-Entropy: 0.08344876 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 24, Accuracy: 1, Cross-Entropy: 0.08278403 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 25, Accuracy: 1, Cross-Entropy: 0.0821895 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 26, Accuracy: 1, Cross-Entropy: 0.08161344 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 27, Accuracy: 1, Cross-Entropy: 0.08109526 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 28, Accuracy: 1, Cross-Entropy: 0.08059177 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 29, Accuracy: 1, Cross-Entropy: 0.08013667 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 30, Accuracy: 1, Cross-Entropy: 0.07969329 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 31, Accuracy: 1, Cross-Entropy: 0.0792909 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 32, Accuracy: 1, Cross-Entropy: 0.07889795 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 33, Accuracy: 1, Cross-Entropy: 0.07854007 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 34, Accuracy: 1, Cross-Entropy: 0.07818998 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 35, Accuracy: 1, Cross-Entropy: 0.07787006 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 36, Accuracy: 1, Cross-Entropy: 0.07755657 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 37, Accuracy: 1, Cross-Entropy: 0.07726936 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 38, Accuracy: 1, Cross-Entropy: 0.07698746 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 39, Accuracy: 1, Cross-Entropy: 0.07672852 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 40, Accuracy: 1, Cross-Entropy: 0.07647407 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 41, Accuracy: 1, Cross-Entropy: 0.07623982 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 42, Accuracy: 1, Cross-Entropy: 0.07600942 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 43, Accuracy: 1, Cross-Entropy: 0.07579684 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 44, Accuracy: 1, Cross-Entropy: 0.07558744 Phase: Training, Dataset used: Validation, Batch Processed Count: 2, Epoch: 45, Accuracy: 1, Cross-Entropy: 0.07539399 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_0\custom_retrained_model_based_on_resnet_v2_101_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationEarlyStopping with memory usage 4,127,510,528.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowStringTest 2026-04-14 06:32:36.311977: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: model_string_test 2026-04-14 06:32:36.313031: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.313260: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: model_string_test 2026-04-14 06:32:36.313943: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 1969 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowStringTest with memory usage 4,127,608,832.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowSaveAndLoadSavedModel 2026-04-14 06:32:36.349392: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:36.351603: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.351958: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:36.359781: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:36.388624: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 39232 microseconds. 2026-04-14 06:32:36.608796: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: C:\h\w\9D2C08F9\t\TensorFlowTransformer_c03ab0de-9041-4216-82c3-b7aa40e2025e 2026-04-14 06:32:36.610900: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.611225: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: C:\h\w\9D2C08F9\t\TensorFlowTransformer_c03ab0de-9041-4216-82c3-b7aa40e2025e 2026-04-14 06:32:36.618687: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:36.647040: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 38240 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowSaveAndLoadSavedModel with memory usage 4,132,024,320.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowGettingSchemaMultipleTimes 2026-04-14 06:32:36.726760: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:36.728551: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.728780: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:36.734398: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:36.754431: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 27665 microseconds. 2026-04-14 06:32:36.772063: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:36.774061: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.774388: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:36.781173: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:36.809060: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 36994 microseconds. 2026-04-14 06:32:36.821083: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:36.822376: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.822595: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:36.827349: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:36.846725: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 25636 microseconds. 2026-04-14 06:32:36.872413: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:36.874411: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:36.874659: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:36.879820: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:36.911579: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 39160 microseconds. 2026-04-14 06:32:37.072447: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:37.074286: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:37.074590: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:37.081397: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:37.107835: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 35384 microseconds. 2026-04-14 06:32:37.139910: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:37.142458: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:37.142681: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:37.147589: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:37.167725: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 27816 microseconds. 2026-04-14 06:32:37.204091: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:37.205993: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:37.206319: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:37.212749: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:37.238215: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 34117 microseconds. 2026-04-14 06:32:37.251682: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:37.253221: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:37.253474: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:37.258684: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:37.279681: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 27992 microseconds. 2026-04-14 06:32:37.296218: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:37.298062: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:37.298372: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:37.304825: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:37.330253: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 34031 microseconds. 2026-04-14 06:32:37.345552: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:32:37.347003: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:32:37.347225: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:32:37.352284: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:32:37.372794: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 27237 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowGettingSchemaMultipleTimes with memory usage 1,809,543,168.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationWithPolynomialLRScheduling Saver not created because there are no variables in the graph to restore Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 0, Accuracy: 0.54, Cross-Entropy: 1.342551, Learning Rate: 0.00505 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 0, Accuracy: 0.8888889, Cross-Entropy: 0.5382147 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 1, Accuracy: 0.82, Cross-Entropy: 0.7000598, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 1, Accuracy: 0.8888889, Cross-Entropy: 0.4742242 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 2, Accuracy: 0.9399999, Cross-Entropy: 0.5676862, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 2, Accuracy: 0.8888889, Cross-Entropy: 0.472762 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 3, Accuracy: 0.9399999, Cross-Entropy: 0.5646911, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 3, Accuracy: 0.8888889, Cross-Entropy: 0.4713005 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 4, Accuracy: 0.9399999, Cross-Entropy: 0.5617303, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 4, Accuracy: 0.8888889, Cross-Entropy: 0.4698397 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 5, Accuracy: 0.9399999, Cross-Entropy: 0.5588027, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 5, Accuracy: 0.8888889, Cross-Entropy: 0.4683802 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 6, Accuracy: 0.9399999, Cross-Entropy: 0.5559078, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 6, Accuracy: 0.8888889, Cross-Entropy: 0.4669221 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 7, Accuracy: 0.9399999, Cross-Entropy: 0.5530447, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 7, Accuracy: 0.8888889, Cross-Entropy: 0.4654658 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 8, Accuracy: 0.9399999, Cross-Entropy: 0.5502126, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 8, Accuracy: 0.8888889, Cross-Entropy: 0.4640115 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 9, Accuracy: 0.9399999, Cross-Entropy: 0.5474107, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 9, Accuracy: 0.8888889, Cross-Entropy: 0.4625594 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 10, Accuracy: 0.9399999, Cross-Entropy: 0.5446385, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 10, Accuracy: 0.8888889, Cross-Entropy: 0.4611099 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 11, Accuracy: 0.9399999, Cross-Entropy: 0.5418953, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 11, Accuracy: 0.8888889, Cross-Entropy: 0.4596632 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 12, Accuracy: 0.9399999, Cross-Entropy: 0.5391804, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 12, Accuracy: 0.8888889, Cross-Entropy: 0.4582195 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 13, Accuracy: 0.96, Cross-Entropy: 0.5364932, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 13, Accuracy: 0.8888889, Cross-Entropy: 0.4567791 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 14, Accuracy: 0.96, Cross-Entropy: 0.5338333, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 14, Accuracy: 0.8888889, Cross-Entropy: 0.4553421 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 15, Accuracy: 0.96, Cross-Entropy: 0.5311998, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 15, Accuracy: 0.8888889, Cross-Entropy: 0.4539089 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 16, Accuracy: 0.96, Cross-Entropy: 0.5285925, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 16, Accuracy: 0.8888889, Cross-Entropy: 0.4524795 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 17, Accuracy: 0.96, Cross-Entropy: 0.5260107, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 17, Accuracy: 0.8888889, Cross-Entropy: 0.4510542 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 18, Accuracy: 0.96, Cross-Entropy: 0.5234541, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 18, Accuracy: 0.8888889, Cross-Entropy: 0.4496332 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 19, Accuracy: 0.96, Cross-Entropy: 0.5209219, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 19, Accuracy: 0.8888889, Cross-Entropy: 0.4482168 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 20, Accuracy: 0.96, Cross-Entropy: 0.518414, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 20, Accuracy: 0.8888889, Cross-Entropy: 0.4468048 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 21, Accuracy: 0.96, Cross-Entropy: 0.5159298, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 21, Accuracy: 0.8888889, Cross-Entropy: 0.4453976 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 22, Accuracy: 0.96, Cross-Entropy: 0.5134688, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 22, Accuracy: 0.8888889, Cross-Entropy: 0.4439955 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 23, Accuracy: 0.96, Cross-Entropy: 0.5110307, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 23, Accuracy: 0.8888889, Cross-Entropy: 0.4425983 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 24, Accuracy: 0.96, Cross-Entropy: 0.5086152, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 24, Accuracy: 0.8888889, Cross-Entropy: 0.4412064 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 25, Accuracy: 0.96, Cross-Entropy: 0.5062217, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 25, Accuracy: 0.8888889, Cross-Entropy: 0.4398198 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 26, Accuracy: 0.96, Cross-Entropy: 0.50385, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 26, Accuracy: 0.8888889, Cross-Entropy: 0.4384387 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 27, Accuracy: 0.96, Cross-Entropy: 0.5014996, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 27, Accuracy: 0.8888889, Cross-Entropy: 0.4370632 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 28, Accuracy: 0.96, Cross-Entropy: 0.4991702, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 28, Accuracy: 0.8888889, Cross-Entropy: 0.4356934 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 29, Accuracy: 0.96, Cross-Entropy: 0.4968616, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 29, Accuracy: 0.8888889, Cross-Entropy: 0.4343293 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 30, Accuracy: 0.96, Cross-Entropy: 0.4945733, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 30, Accuracy: 0.8888889, Cross-Entropy: 0.4329711 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 31, Accuracy: 0.96, Cross-Entropy: 0.4923051, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 31, Accuracy: 0.8888889, Cross-Entropy: 0.431619 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 32, Accuracy: 0.96, Cross-Entropy: 0.4900567, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 32, Accuracy: 0.8888889, Cross-Entropy: 0.4302729 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 33, Accuracy: 0.96, Cross-Entropy: 0.4878276, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 33, Accuracy: 0.8888889, Cross-Entropy: 0.4289329 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 34, Accuracy: 0.96, Cross-Entropy: 0.4856177, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 34, Accuracy: 0.8888889, Cross-Entropy: 0.4275992 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 35, Accuracy: 0.96, Cross-Entropy: 0.4834268, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 35, Accuracy: 0.8888889, Cross-Entropy: 0.4262718 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 36, Accuracy: 0.96, Cross-Entropy: 0.4812544, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 36, Accuracy: 0.8888889, Cross-Entropy: 0.4249506 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 37, Accuracy: 0.96, Cross-Entropy: 0.4791004, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 37, Accuracy: 0.8888889, Cross-Entropy: 0.4236359 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 38, Accuracy: 0.96, Cross-Entropy: 0.4769644, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 38, Accuracy: 0.8888889, Cross-Entropy: 0.4223276 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 39, Accuracy: 0.96, Cross-Entropy: 0.4748463, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 39, Accuracy: 0.8888889, Cross-Entropy: 0.4210258 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 40, Accuracy: 0.96, Cross-Entropy: 0.4727457, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 40, Accuracy: 0.8888889, Cross-Entropy: 0.4197307 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 41, Accuracy: 0.96, Cross-Entropy: 0.4706625, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 41, Accuracy: 0.8888889, Cross-Entropy: 0.418442 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 42, Accuracy: 0.96, Cross-Entropy: 0.4685963, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 42, Accuracy: 0.8888889, Cross-Entropy: 0.4171599 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 43, Accuracy: 0.96, Cross-Entropy: 0.4665471, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 43, Accuracy: 0.8888889, Cross-Entropy: 0.4158846 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 44, Accuracy: 0.96, Cross-Entropy: 0.4645145, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 44, Accuracy: 0.8888889, Cross-Entropy: 0.4146159 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 45, Accuracy: 0.96, Cross-Entropy: 0.4624983, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 45, Accuracy: 0.8888889, Cross-Entropy: 0.4133539 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 46, Accuracy: 0.96, Cross-Entropy: 0.4604983, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 46, Accuracy: 0.8888889, Cross-Entropy: 0.4120986 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 47, Accuracy: 0.96, Cross-Entropy: 0.4585144, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 47, Accuracy: 0.8888889, Cross-Entropy: 0.41085 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 48, Accuracy: 0.96, Cross-Entropy: 0.4565461, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 48, Accuracy: 0.8888889, Cross-Entropy: 0.4096082 Phase: Training, Dataset used: Train, Batch Processed Count: 5, Epoch: 49, Accuracy: 0.96, Cross-Entropy: 0.4545935, Learning Rate: 0.0001 Phase: Training, Dataset used: Validation, Batch Processed Count: 1, Epoch: 49, Accuracy: 0.8888889, Cross-Entropy: 0.4083732 Saver not created because there are no variables in the graph to restore Restoring parameters from C:\h\w\9D2C08F9\t\bb23ul5e.hka\assets\cached\FPTSUT_0\custom_retrained_model_based_on_resnet_v2_101_299.meta Froze 2 variables. Converted 2 variables to const ops. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassificationWithPolynomialLRScheduling with memory usage 2,019,463,168.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowPrimitiveInputTest 2026-04-14 06:33:54.366462: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: model_primitive_input_test 2026-04-14 06:33:54.367521: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:33:54.367751: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: model_primitive_input_test 2026-04-14 06:33:54.368619: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 2159 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowPrimitiveInputTest with memory usage 2,019,520,512.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTConvSavedModelTest 2026-04-14 06:33:54.382741: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: mnist_model 2026-04-14 06:33:54.384627: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:33:54.384851: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: mnist_model 2026-04-14 06:33:54.388123: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:33:54.427421: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: mnist_model 2026-04-14 06:33:54.431216: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 48472 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformMNISTConvSavedModelTest with memory usage 2,022,678,528.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifarInvalidShape Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifarInvalidShape with memory usage 2,022,690,816.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifarCrossValidationWithInMemoryImages 2026-04-14 06:33:55.325918: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: cifar_saved_model 2026-04-14 06:33:55.327435: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve } 2026-04-14 06:33:55.327655: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: cifar_saved_model 2026-04-14 06:33:55.332620: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle. 2026-04-14 06:33:55.352202: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 26281 microseconds. Finished test: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowTransformCifarCrossValidationWithInMemoryImages with memory usage 2,027,827,200.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestTensorFlow Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestTensorFlow with memory usage 2,044,243,968.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestTensorFlowWithSchema Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestTensorFlowWithSchema with memory usage 2,064,314,368.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestCommandLine 3 columns: a: Vector b: Vector c: Vector Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestCommandLine with memory usage 2,070,183,936.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestLoadMultipleModel Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestLoadMultipleModel with memory usage 2,070,421,504.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestOldSavingAndLoading Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestOldSavingAndLoading with memory usage 2,065,870,848.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestSimpleCase Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TestSimpleCase with memory usage 2,067,333,120.00 and max memory usage 6,362,353,664.00 Starting test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TreatOutputAsBatched Finished test: Microsoft.ML.Tests.TensorFlowEstimatorTests.TreatOutputAsBatched with memory usage 2,102,071,296.00 and max memory usage 6,362,353,664.00 Finished: Microsoft.ML.TensorFlow.Tests Unhandled exception: System.ObjectDisposedException: Cannot access a disposed object. Object name: 'The ThreadLocal object has been disposed.'. at System.Threading.ThreadLocal`1.GetValueSlow() at Tensorflow.BaseSession.DisposeUnmanagedResources(IntPtr handle) at Tensorflow.DisposableObject.Dispose(Boolean disposing) at Tensorflow.DisposableObject.Finalize() System.ObjectDisposedException: Cannot access a disposed object. Object name: 'The ThreadLocal object has been disposed.'. at System.Threading.ThreadLocal`1.GetValueSlow() at Tensorflow.BaseSession.DisposeUnmanagedResources(IntPtr handle) at Tensorflow.DisposableObject.Dispose(Boolean disposing) at Tensorflow.DisposableObject.Finalize() ----- end Tue 04/14/2026 6:33:58.83 ----- exit code 1 ---------------------------------------------------------- Compress-Archive : The path 'C:\h\w\9D2C08F9\w\9AA10905\e\\TestOutput' either does not exist or is not a valid file system path. At line:1 char:1 + Compress-Archive C:\h\w\9D2C08F9\w\9AA10905\e\\TestOutput C:\h\w\9D2C ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (C:\h\w\9D2C08F9\w\9AA10905\e\\TestOutput:String) [Compress-Archive], I nvalidOperationException + FullyQualifiedErrorId : ArchiveCmdletPathNotFound,Compress-Archive ( was unexpected at this time. ['Microsoft.ML.TensorFlow.Tests' END OF WORK ITEM LOG: Command exited with 255]