Reporting and Diagnostics

Ocaramba provides a multi-layered approach to test reporting and diagnostics, ranging from low-level text logs to high-level HTML dashboards and performance metrics. The framework is designed to automatically capture state (screenshots and page source) upon test failure or soft-assertion violations, ensuring that engineers have sufficient data to debug issues without re-running tests.

Diagnostic Capabilities Overview

The diagnostic subsystem integrates several components to provide a comprehensive view of test execution:

• NLog Integration: Structured logging of test lifecycle events and framework operations via NLog Ocaramba.Tests.NUnitExtentReports/Ocaramba.Tests.NUnitExtentReports.csproj12
• Automatic Artifact Capture: Logic within tests and base classes automatically triggers screenshot and page source saving. For example, SaveWebDriverScreenShot captures browser state during execution Ocaramba.Tests.NUnit/Tests/SaveScreenShotsPageSourceTestsNUnit.cs35-43
• ExtentReports Dashboard: A rich HTML reporting interface provided by the Ocaramba.Tests.NUnitExtentReports project Ocaramba.Tests.NUnitExtentReports/Ocaramba.Tests.NUnitExtentReports.csproj26-28
• Performance Tracking: Measurement of execution times for specific blocks of code via PerformanceHelper, with built-in calculations for 90th percentiles and averages OcarambaLite/Helpers/PerformanceHelper.cs36-57

Reporting Architecture

The following diagram illustrates how diagnostic data flows from the test execution layer into various reporting outputs.

Reporting Data Flow

Sources: OcarambaLite/Helpers/PerformanceHelper.cs36-57 Ocaramba.Tests.NUnit/Tests/SaveScreenShotsPageSourceTestsNUnit.cs35-57 Ocaramba.Tests.NUnitExtentReports/Ocaramba.Tests.NUnitExtentReports.csproj12-28

ExtentReports Integration

The Ocaramba.Tests.NUnitExtentReports project provides an advanced reporting implementation using the ExtentReports library Ocaramba.Tests.NUnitExtentReports/Ocaramba.Tests.NUnitExtentReports.csproj26-28 It uses specialized base classes and managers to handle the reporting lifecycle.

• HTML Dashboard: Generates a searchable, filterable dashboard showing test status, execution time, and categories.
• Screenshot Embedding: Automatically attaches failure screenshots directly into the HTML report for immediate visual feedback.
• ExtentTestLogger: A wrapper that allows developers to send custom log messages (Info, Pass, Fail, Warning) directly to the Extent report.

For details, see ExtentReports Integration (NUnitExtentReports).

ExtentReports Entity Mapping

This diagram maps the logical reporting concepts to the specific classes and methods used in the implementation.

ExtentReports Code Mapping

Sources: Ocaramba.Tests.NUnitExtentReports/Ocaramba.Tests.NUnitExtentReports.csproj26-28 Ocaramba.Tests.NUnitExtentReports/packages.lock.json5-15

Performance Measurement and Reporting

Ocaramba includes a built-in performance subsystem via the PerformanceHelper class OcarambaLite/Helpers/PerformanceHelper.cs36-37 This system allows for granular timing of business processes within tests.

• PerformanceHelper: Provides StartMeasure() and StopMeasure(string title) methods to track execution durations OcarambaLite/Helpers/PerformanceHelper.cs99-120 It maintains a loadTimeList of measured times OcarambaLite/Helpers/PerformanceHelper.cs48-49
• Percentile Calculation: The AllGroupedDurationsMilliseconds property automatically calculates the 90th percentile and average duration for grouped scenarios OcarambaLite/Helpers/PerformanceHelper.cs68-88
• CI Integration: PrintPerformanceResultsHelper formats these metrics into service messages compatible with TeamCity (##teamcity[testFinished ...]) and AppVeyor, allowing performance trends to be tracked directly in the CI/CD pipeline OcarambaLite/Helpers/PrintPerformanceResultsHelper.cs44-102

Sources: OcarambaLite/Helpers/PerformanceHelper.cs36-121 OcarambaLite/Helpers/PrintPerformanceResultsHelper.cs36-102

Artifact Capture and Logging

The framework ensures that test execution is transparent through logging and artifact generation.

Artifact Capture

Tests can explicitly or automatically capture browser state:

1. Screenshots: SaveWebDriverScreenShot() captures the current browser view and saves it to the ScreenShotFolder Ocaramba.Tests.NUnit/Tests/SaveScreenShotsPageSourceTestsNUnit.cs35-43
2. Page Source: SaveSourcePage() captures the HTML source of the current page and saves it to the PageSourceFolder Ocaramba.Tests.NUnit/Tests/SaveScreenShotsPageSourceTestsNUnit.cs47-56
3. JavaScript Errors: The framework can check for and log browser console errors during execution Ocaramba.UnitTests/Tests/JavaScriptAlertsTestsNUnit.cs35-44

Logging and File Comparison

• NLog: Used throughout the framework to log errors and informational messages. Classes like Select and Table use Logger.Error or Logger.Debug to report interaction details OcarambaLite/WebElements/Select.cs94-96 OcarambaLite/WebElements/Table.cs78
• File Diagnostics: The framework includes helpers for comparing downloaded files (CSV, TXT) against baseline "live" files, logging discrepancies when they occur Ocaramba.Tests.NUnit/Tests/CompareFilesDataDrivenTests.cs48-93
• Naming Helpers: NameHelper provides diagnostics-friendly utilities like ShortenFileName to ensure artifact paths do not exceed OS limits OcarambaLite/Helpers/NameHelper.cs72-90

Sources: Ocaramba.Tests.NUnit/Tests/SaveScreenShotsPageSourceTestsNUnit.cs35-57 OcarambaLite/Helpers/PerformanceHelper.cs117 OcarambaLite/WebElements/Select.cs39-40 OcarambaLite/Helpers/NameHelper.cs34-109