This is a 2 part-series post.

  • Part 1: Key concepts that you should know when using OpenTelemetry Metrics with .NET. If you want to read it, click here.
  • Part 2: A practical example about how to add OpenTelemetry Metrics on a real life .NET app and how to visualize those metrics using Prometheus and Grafana.

Just show me the code!
As always, if you don’t care about the post I have uploaded the source code on my Github.

In part 1 we talked about some OpenTelemetry Metrics key concepts, now it’s time to focus on instrumenting a real application.

This is what we are going to build.

otel-metrics-app-diagram

  • The BookStore WebAPI will generate some business metrics and use the OTLP exporter package (OpenTelemetry.Exporter.OpenTelemetryProtocol) to send the metric data to the OpenTelemetry Collector.
  • Prometheus will obtain the metric data from the OpenTelemetry Collector.
  • We will have a Grafana dashboard to visualize the metrics emitted by the BookStore WebApi.

Application

The application we’re going to instrument using OpenTelemetry Metrics is a BookStore API built using .NET 7.
The API can do the following actions:

  • Get, add, update and delete book categories.
  • Get, add, update and delete books.
  • Get, add, update and delete inventory.
  • Get, add and update orders.

For a better understanding, here’s how the database diagram looks like:

otel-metrics-bookstore-database-diagram

OpenTelemetry Metrics

The first step before writing any code is to decide what we want to measure and what kind of instruments are we going to use.

BookStore API custom metrics

The following business metrics will be instrumented directly on the application using the Metrics API.

  • BooksAddedCounter is a Counter that counts books added to the store.
  • BooksDeletedCounter is a Counter that counts books deleted from the store.
  • BooksUpdatedCounter is a Counter that counts books updated.
  • TotalBooksUpDownCounter is an UpDownCounter that contains the total number of books that the store has available at any given time.
  • CategoriesAddedCounter is a Counter that counts book categories added to the store.
  • CategoriesDeletedCounter is a Counter that counts book categories deleted from the store.
  • CategoriesUpdatedCounter is a Counter that counts book categories updated.
  • TotalCategoriesGauge is an ObservableGauge that contains the total number of book categories that the store has at any given time
  • OrdersPriceHistogram is a Histogram that records the price distribution of the orders.
  • NumberOfBooksPerOrderHistogram is a Histogram that records the number of books distribution per order.
  • OrdersCanceledCounter is an ObservableCounter that counts the total number of orders cancelled.
  • TotalOrdersCounter is a Counter that counts the total number of orders that the store has received.

Http requests metrics

Those metrics are generated by the OpenTelemetry.Instrumentation.AspNetCore NuGet package.
This is an instrumentation library, which instruments .NET and creates metrics about incoming web requests.

There is no need to instrument anything on the application. To start using the OpenTelemetry.Instrumentation.AspNetCore package you only need to add the AddAspNetCoreInstrumentation() extension method when setting up the .NET OpenTelemetry MeterProvider component. Here’s an example:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
    .AddAspNetCoreInstrumentation()
);

System.Runtime performance metrics

Those metrics are generated by the OpenTelemetry.Instrumentation.Runtime NuGet package. This is an instrumentation library, which instruments .NET Runtime and collects runtime performance metrics.

There is no need to instrument anything on the application. To start using the OpenTelemetry.Instrumentation.Runtime package you only need to add the AddRuntimeInstrumentation() extension method when setting up the .NET OpenTelemetry MeterProvider component. Here’s an example:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
    .AddRuntimeInstrumentation()
);

The OpenTelemetry.Instrumentation.Runtime package collects telemetry about the following System.Runtime counters:

  • process.runtime.dotnet.gc.collections.count: Number of garbage collections that have occurred since process start.
  • process.runtime.dotnet.gc.objects.size: Count of bytes currently in use by objects in the GC heap that haven’t been collected yet. Fragmentation and other GC committed memory pools are excluded.
  • process.runtime.dotnet.gc.allocations.size: Count of bytes allocated on the managed GC heap since the process start
  • process.runtime.dotnet.gc.committed_memory.size: The amount of committed virtual memory for the managed GC heap, as observed during the latest garbage collection.
  • process.runtime.dotnet.gc.heap.size: The heap size (including fragmentation), as observed during the latest garbage collection.
  • process.runtime.dotnet.gc.heap.fragmentation.size: The heap fragmentation, as observed during the latest garbage collection.
  • process.runtime.dotnet.jit.il_compiled.size: Count of bytes of intermediate language that have been compiled since the process start.
  • process.runtime.dotnet.jit.methods_compiled.count: The number of times the JIT compiler compiled a method since the process start.
  • process.runtime.dotnet.jit.compilation_time: The amount of time the JIT compiler has spent compiling methods since the process start.
  • process.runtime.dotnet.monitor.lock_contention.count: The number of times there was contention when trying to acquire a monitor lock since the process start.
  • process.runtime.dotnet.thread_pool.threads.count: The number of thread pool threads that currently exist.
  • process.runtime.dotnet.thread_pool.completed_items.count: The number of work items that have been processed by the thread pool since the process start.
  • process.runtime.dotnet.thread_pool.queue.length: The number of work items that are currently queued to be processed by the thread pool.
  • process.runtime.dotnet.timer.count: The number of timer instances that are currently active.
  • process.runtime.dotnet.assemblies.count: The number of .NET assemblies that are currently loaded.
  • process.runtime.dotnet.exceptions.count: Count of exceptions that have been thrown in managed code, since the observation started.

Some of the GC related metrics will be unavailable until at least one garbage collection has occurred.

Process metrics

Those metrics are generated by the OpenTelemetry.Instrumentation.Process NuGet package. This is an Instrumentation Library, which instruments .NET and collects telemetry about the running process.

To start using the OpenTelemetry.Instrumentation.Process package you only need to add the AddProcessInstrumentation() extension method when setting up the .NET OpenTelemetry component. Here’s an example:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
        .AddProcessInstrumentation()
);

The OpenTelemetry.Instrumentation.Process package collects the following metrics of the running process:

  • process.memory.usage: The amount of physical memory allocated for this process.
  • process.memory.virtual: The amount of committed virtual memory for this process. One way to think of this is all the address space this process can read from without triggering an access violation; this includes memory backed solely by RAM, by a swapfile/pagefile and by other mapped files on disk.
  • process.cpu.time: Total CPU seconds broken down by states.
  • process.cpu.count: The number of processors (CPU cores) available to the current process.
  • process.threads: Process threads count.

OpenTelemetry .NET Client

To get started with OpenTelemetry Metrics we’re going to need the following packages.

<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.5.0-alpha.2" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.14" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.1.0-rc.2" />
<PackageReference Include="OpenTelemetry.Instrumentation.Process" Version="0.5.0-beta.2" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.5.0-alpha.2" />
  • The OpenTelemetry.Extensions.Hosting package contains some extensions that allows us to configure the MeterProvider.
  • The OpenTelemetry.Instrumentation.* packages are instrumentation libraries. These packages are instrumenting common libraries/functionalities/classes so we don’t have to do all the heavy lifting by ourselves. In our application we’re using the following ones:
    • The OpenTelemetry.Instrumentation.AspNetCore package generats and collects metrics about incoming web requests.
    • The OpenTelemetry.Instrumentation.Runtime package collects runtime performance metrics.
    • The OpenTelemetry.Instrumentation.Process package collects process metrics.
  • The OpenTelemetry.Exporter.OpenTelemetryProtocol package allows us to export the metrics to the OpenTelemetry Collector using the OTLP protocol.

Add OpenTelemetry Metrics on the BookStore app

1 - Setup the MeterProvider

var meters = new OtelMetrics();

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("BookStore.WebApi"))
    .AddMeter(meters.MetricName)
    .AddAspNetCoreInstrumentation()
    .AddRuntimeInstrumentation()
    .AddProcessInstrumentation()
    .AddOtlpExporter(opts =>
    {
        opts.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]);
    }));   

Let’s review what we’re doing line by line.

AddOpenTelemetry().WithMetrics()

OpenTelemetry Metrics works by using the MeterProvider to create a Meter and associating it with one or more Instruments, each of which is used to create a series of Measurements.

The MeterProvider holds the configuration for metrics like Meter names, Readers or Views. The MeterProvider must be configured using the .WithMetrics() extension method.

The AddOpenTelemetry() is responsible for registering an IHostedService that starts the tracing and metric services.

SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("BookStore.WebApi"))

A Resource is the immutable representation of the entity producing the telemetry. With the SetResourceBuilder method we’re configuring the Resource for the application.

The SetResourceBuilder gives us the possibility to configure attributes like the service name or the application name amongst others.

Using the AddService("BookStore.Webapi") method we can set the service name as an attribute of the metric data. For example, if we take a look at the OrdersCanceledCounter instrument on Prometheus, we will see the following information:

otel-metrics-prometheus-resource-builder

AddMeter(meters.MetricName)

Any Instrument that we create in our application needs to be associated with a Meter. The AddMeter() extension method configures OpenTelemetry to transmit all the metrics collected by this concrete Meter.

As you’ll see later, in the BookStore app I have a single Meter with multiple Instruments on it, but you can also have multiple Meters in a single application, in that case you’ll need to add multiple calls to the AddMeter() method.

AddAspNetCoreInstrumentation()

This method comes from the OpenTelemetry.Instrumentation.AspNetCore NuGet package, it instruments .NET and collects metrics and traces about incoming web requests.

AddRuntimeInstrumentation()

This method comes from the OpenTelemetry.Instrumentation.Runtime NuGet package, it instruments .NET and collects runtime performance metrics.

AddProcessInstrumentation()

This method comes from the OpenTelemetry.Instrumentation.Process NuGet package, it instruments .NET and collects process related metrics like CPU or memory.

AddOtlpExporter(opts =>
{
    opts.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]);
}));

The AddOtlpExporter method is used to configure the exporter that sends all the metric data to the OpenTelemetry Collector.

2 - Create the Meter and Instruments

After setting up the MeterProvider, it’s time to create a Meter and use it to create Instruments.

There are a few ways to do that, but I’m going to show you how I tend to do it.
First of all, I’m going to create a Singleton class which will contain:

  • A meter.
  • The instruments associated with the meter.
  • A series of helper methods to record measurements with those instruments.

Here’s how the class looks like:

using System.Collections.Generic;
using System.Diagnostics.Metrics;

namespace BookStore.Infrastructure.Metrics
{
    public class OtelMetrics
    {
        //Books meters
        private  Counter<int> BooksAddedCounter { get; }
        private  Counter<int> BooksDeletedCounter { get; }
        private  Counter<int> BooksUpdatedCounter { get; }
        private  UpDownCounter<int> TotalBooksUpDownCounter { get; }

        //Categories meters
        private Counter<int> CategoriesAddedCounter { get; }
        private Counter<int> CategoriesDeletedCounter { get; }
        private Counter<int> CategoriesUpdatedCounter { get; }
        private ObservableGauge<int> TotalCategoriesGauge { get; }
        private int _totalCategories = 0;

        //Order meters
        private Histogram<double> OrdersPriceHistogram { get; }
        private Histogram<int> NumberOfBooksPerOrderHistogram { get; }
        private ObservableCounter<int> OrdersCanceledCounter { get; }
        private int _ordersCanceled = 0;
        private Counter<int> TotalOrdersCounter { get; }

        public string MetricName { get; }

        public OtelMetrics(string meterName = "BookStore")
        {
            var meter = new Meter(meterName);
            MetricName = meterName;

            BooksAddedCounter = meter.CreateCounter<int>("books-added", "Book");
            BooksDeletedCounter = meter.CreateCounter<int>("books-deleted", "Book");
            BooksUpdatedCounter = meter.CreateCounter<int>("books-updated", "Book");
            TotalBooksUpDownCounter = meter.CreateUpDownCounter<int>("total-books", "Book");
            
            CategoriesAddedCounter = meter.CreateCounter<int>("categories-added", "Category");
            CategoriesDeletedCounter = meter.CreateCounter<int>("categories-deleted", "Category");
            CategoriesUpdatedCounter = meter.CreateCounter<int>("categories-updated", "Category");
            TotalCategoriesGauge = meter.CreateObservableGauge<int>("total-categories", () => _totalCategories);

            OrdersPriceHistogram = meter.CreateHistogram<double>("orders-price", "Euros", "Price distribution of book orders");
            NumberOfBooksPerOrderHistogram = meter.CreateHistogram<int>("orders-number-of-books", "Books", "Number of books per order");
            OrdersCanceledCounter = meter.CreateObservableCounter<int>("orders-canceled", () => _ordersCanceled);
            TotalOrdersCounter = meter.CreateCounter<int>("total-orders", "Orders");
        }

        //Books meters
        public void AddBook() => BooksAddedCounter.Add(1);
        public void DeleteBook() => BooksDeletedCounter.Add(1);
        public void UpdateBook() => BooksUpdatedCounter.Add(1);
        public void IncreaseTotalBooks() => TotalBooksUpDownCounter.Add(1);
        public void DecreaseTotalBooks() => TotalBooksUpDownCounter.Add(-1);

        //Categories meters
        public void AddCategory() => CategoriesAddedCounter.Add(1);
        public void DeleteCategory() => CategoriesDeletedCounter.Add(1);
        public void UpdateCategory() => CategoriesUpdatedCounter.Add(1);
        public void IncreaseTotalCategories() => _totalCategories++;
        public void DecreaseTotalCategories() => _totalCategories--;

        //Orders meters
        public void RecordOrderTotalPrice(double price) => OrdersPriceHistogram.Record(price);
        public void RecordNumberOfBooks(int amount) => NumberOfBooksPerOrderHistogram.Record(amount);
        public void IncreaseOrdersCanceled() => _ordersCanceled++;
        public void IncreaseTotalOrders(string city) => TotalOrdersCounter.Add(1, KeyValuePair.Create<string, object>("City", city));

    }
}

In the class constructor, we create the meter and then use it to create every necessary instrument. Also we create a series of public helper methods to record measurements.

  • Why create all these helper methods (AddBook, DeleteBook, UpdateBook, etcs)?

To improve code readability, it is easier to understand what this line of code _meters.AddBook() is doing, rather than this other one BooksAddedCounter.Add(1).

3 - Record measurements using the instruments

Now it’s time to use the instruments we have created in the previous section to start recording measurements.

You just need to inject the OtelMetrics instance whenever we want to record a measurement and use any of the helper methods exposed on the OtelMetrics class.

Record book metrics

  • Every time a new book gets added into the database.
    • Increase +1 the BooksAddedCounter instrument and increase +1 the TotalBooksUpDownCounter instrument.
  • Every time a new book gets updated.
    • Increase +1 the BooksUpdatedCounter instrument.
  • Every time a new book gets deleted from the database.
    • Increase +1 the BooksDeletedCounter instrument and decrease -1 the TotalBooksUpDownCounter instrument.

The next snippet of code shows how to record those measurements every time a book gets added, updated or deleted from the database.

public class BookRepository : Repository<Book>, IBookRepository
{
    private readonly OtelMetrics _meters;

    public BookRepository(
        BookStoreDbContext context, 
        OtelMetrics meters) : base(context)
    {
        _meters = meters;
    }

    public override async Task<List<Book>> GetAll()
    {
        return await Db.Books.Include(b => b.Category)
            .OrderBy(b => b.Name)
            .ToListAsync();
    }

    public override async Task<Book> GetById(int id)
    {
        return await Db.Books.Include(b => b.Category)
            .Where(b => b.Id == id)
            .FirstOrDefaultAsync();
    }

    public async Task<IEnumerable<Book>> GetBooksByCategory(int categoryId)
    {
        return await Search(b => b.CategoryId == categoryId);
    }

    public async Task<IEnumerable<Book>> SearchBookWithCategory(string searchedValue)
    {
        return await Db.Books.AsNoTracking()
            .Include(b => b.Category)
            .Where(b => b.Name.Contains(searchedValue) || 
                        b.Author.Contains(searchedValue) ||
                        b.Description.Contains(searchedValue) ||
                        b.Category.Name.Contains(searchedValue))
            .ToListAsync();
    }

    public override async Task Add(Book entity)
    {
        await base.Add(entity);

        _meters.AddBook();
        _meters.IncreaseTotalBooks();
    }

    public override async Task Update(Book entity)
    {
        await base.Update(entity);

        _meters.UpdateBook();
    }

    public override async Task Remove(Book entity)
    {
        await base.Remove(entity);

        _meters.DeleteBook();
        _meters.DecreaseTotalBooks();
    }
}

Record book categories metrics

  • Every time a new book category gets added into the database.
    • Increase +1 the CategoriesAddedCounter instrument and increase +1 the TotalCategoriesGauge instrument.
  • Every time a new book category gets updated.
    • Increase +1 the CategoriesUpdatedCounter instrument.
  • Every time a new book category gets deleted from the database.
    • Increase +1 the CategoriesDeletedCounter instrument and decrease -1 the TotalCategoriesGauge instrument.

The next snippet of code shows how to record those measurements every time a book category gets added, updated or deleted from the database.

public class CategoryRepository : Repository<Category>, ICategoryRepository
{
    private readonly OtelMetrics _meters;

    public CategoryRepository(BookStoreDbContext context, 
        OtelMetrics meters) : base(context)
    {
        _meters = meters;
    }

    public override async Task Add(Category entity)
    {
        await base.Add(entity);
        _meters.AddCategory();
        _meters.IncreaseTotalCategories();
    }

    public override async Task Update(Category entity)
    {
        await base.Update(entity);
        _meters.UpdateCategory();
    }

    public override async Task Remove(Category entity)
    {
        await base.Remove(entity);
        _meters.DeleteCategory();
        _meters.DecreaseTotalCategories();
    }
}

Record orders metrics

  • Every time a new order gets added into the database.
    • Increase +1 the TotalOrdersCounter instrument.
    • Record the order total price using the OrdersPriceHistogram instrument.
    • Record the amount of books in the order using the NumberOfBooksPerOrderHistogram instrument.
  • Every time a new order gets updated.
    • Increase +1 the OrdersCanceledCounter instrument.

The next snippet of code shows how to record those measurements every time an order gets added or updated.

public class OrderRepository : Repository<Order>, IOrderRepository
{
    private readonly OtelMetrics _meters;

    public OrderRepository(BookStoreDbContext context, OtelMetrics meters) 
        : base(context)
    {
        _meters = meters;
    }

    public override async Task<Order> GetById(int id)
    {
        return await Db.Orders
            .Include(b => b.Books)
            .FirstOrDefaultAsync(x => x.Id == id);
    }

    public override async Task<List<Order>> GetAll()
    {
        return await Db.Orders
            .Include(b => b.Books)
            .ToListAsync();
    }

    public override async Task Add(Order entity)
    {
        DbSet.Add(entity);
        await base.SaveChanges();

        _meters.RecordOrderTotalPrice(entity.TotalAmount);
        _meters.RecordNumberOfBooks(entity.Books.Count);
        _meters.IncreaseTotalOrders(entity.City);
    }

    public override async Task Update(Order entity)
    {
        await base.Update(entity);

        _meters.IncreaseOrdersCanceled();
    }

    public async Task<List<Order>> GetOrdersByBookId(int bookId)
    {
        return await Db.Orders.AsNoTracking()
            .Include(b => b.Books)
            .Where(x => x.Books.Any(y => y.Id == bookId))
            .ToListAsync();

    }
}

As you can see from the snippets of code from this section recording a measurement it’s a really simple task, just invoke the instrument function wherever you want to record a measurement, and that’s it!

4 - Setup the Histogram bucket aggregation accordingly

A Histogram is a graphical representation of the distribution of numerical data. It groups values into buckets and then counts how many values fall into each bucket.

When using a Histogram instrument, it’s important to make sure the buckets are also configured properly. The bucket histogram aggregation default values are [ 0, 5, 10, 25, 50, 75, 100, 250, 500, 1000 ], and that’s not always ideal.

In the BookStore app we are using 2 Histograms:

  • OrdersPriceHistogram: Shows the price distribution of the orders.
  • NumberOfBooksPerOrderHistogram: Shows the number of books per order distribution.

For the NumberOfBooksPerOrderHistogram makes no sense using the bucket aggregation default values because no one is going to make an order that contains 250, 500 or 1000 books. And the same could be said for the OrdersPriceHistogram.

To customize the bucket aggregation values accordingly to every Histogram we need to use a View.
A View in OpenTelemetry defines an aggregation, which takes a series of measurements and expresses them as a single metric value at that point in time.

To create a View we can use the AddView extension method from the MeterProvider. Like this:

builder.Services.AddOpenTelemetry().WithMetrics(opts => opts
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("BookStore.WebApi"))
    .AddMeter(meters.MetricName)
    .AddAspNetCoreInstrumentation()
    .AddRuntimeInstrumentation()
    .AddProcessInstrumentation()
    .AddView(
        instrumentName: "orders-price",
        new ExplicitBucketHistogramConfiguration { Boundaries = new double[] { 15, 30, 45, 60, 75 } })
    .AddView(
        instrumentName: "orders-number-of-books",
        new ExplicitBucketHistogramConfiguration { Boundaries = new double[] { 1, 2, 5 } })
    .AddOtlpExporter(options  =>
    {
        options.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"] 
                                   ?? throw new InvalidOperationException());
    }));

OpenTelemetry Collector

The OpenTelemetry Collector consists of three components:

  • Receivers: Can be push or pull based, is how data gets into the Collector.
  • Processors: Run on data between being received and being exported.
  • Exporters: Can be push or pull based, is how you send data to one or more backends/destinations.

In this case, the OpenTelemetry Collector receives metrics from the BookStore API via gRPC and exports them into Prometheus.
Here’s how the OTEL Collector config file looks like:

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

processors:
  batch:

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Prometheus

Prometheus must be configured to scrape the OpenTelemetry Collector metrics endpoints.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 5s
    static_configs:
      - targets: ['otel-collector:8889']
      - targets: ['otel-collector:8888']

After setting up Prometheus, we are going to generate traffic on the BookStore API and afterwards access Prometheus to start analyzing the metrics that the app is sending us.

The business metrics from the BookStore API are all available in Prometheus.

otel-metrics-prometheus-metrics-explorer

If we take a peek at the “orders-price” Histogram we can see that the bucket aggregation values are the correct ones that we defined in the MeterProvider using the AddView extension method.

otel-metrics-prometheus-orders-price-histogram.png

The metrics about incoming requests generated by the OpenTelemetry.Instrumentation.AspNetCore NuGet package are also available on Prometheus.

otel-metrics-prometheus-incoming-requests-metrics.png

The metrics generated by the OpenTelemetry.Instrumentation.Runtime and the OpenTelemetry.Instrumentation.Process packages are also being ingested by Prometheus.

otel-metrics-prometheus-perf-counter-and-process-metrics

Grafana

Having the metric data from the BookStore API in Prometheus is great at all, but we need to visualize them in a friendly manner, so let’s build a few Grafana dashboards.

Not going to explain how to build a Grafana dashboard, that’s out of the scope for this post.

I ended up building 3 dashboards.

Custom metrics dashboard

This dashboard uses the business metrics instrumented directly on the application using the Metrics API.

otel-metrics-bookstore-custom-metrics

Here’s a closer look of how the “Orders” panel from the dashboard look like:

otel-metrics-bookstore-custom-orders-metrics

Http Requests dashboard

This dashboard uses the metrics generated by the OpenTelemetry.Instrumentation.AspNetCore NuGet package.

otel-metrics-http-requests-dashboard

Performance counters & process metrics dashboard

This dashboard uses metrics generated by the OpenTelemetry.Instrumentation.Runtime and the OpenTelemetry.Instrumentation.Process packages.

otel-metrics-runtime-perf-counters-and-process-dashboard

How to test the BookStore Api

If you want to take a look at the source code, you can go to my GitHub repository.

If you want to execute the BookStore API for yourselves, I have uploaded a docker-compose file that starts up the app and also the external dependencies.
The external dependencies (Prometheus, MSSQL Server, Grafana and OpenTelemetry Collector) are already preconfigured so you don’t need to do any extra setup. Just run docker-compose up and you’re good to go!

Here’s how the docker-compose file looks like:

version: '3.8'

networks:
  metrics:
    name: bookstore-network
    
services:
  mssql:
    build: 
      context: ./scripts/sql
    ports:
      - "1433:1433"  
    environment:
      SA_PASSWORD: "P@ssw0rd?"
      ACCEPT_EULA: "Y"
    networks:
      - metrics

  prometheus:
    build: 
      context: ./scripts/prometheus
    depends_on:
      - app
    ports:
      - 9090:9090
    networks:
      - metrics

  grafana:
    build: 
      context: ./scripts/grafana
    depends_on:
      - prometheus
    ports:
      - 3000:3000
    networks:
      - metrics
  
  otel-collector:
    image: otel/opentelemetry-collector:0.73.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./scripts/otel-collector/otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "8888:8888" 
      - "8889:8889" 
      - "13133:13133"
      - "4317:4317"
    networks:
      - metrics

  app:
    build:
      context: ./
      dockerfile: ./src/BookStore.WebApi/Dockerfile
    depends_on:
      - mssql
      - otel-collector
    ports:
      - 5001:8080
    environment:
      ConnectionStrings__DbConnection: Server=mssql;Database=BookStore;User Id=SA;Password=P@ssw0rd?;Encrypt=False
      Otlp__Endpoint: http://otel-collector:4317
    networks:
      - metrics

How to generate metrics to test the Grafana dashboards

In my GitHub repository you’ll also find a seed-data.sh Shell script, this script will invoke some endpoints of the BookStore API via cURL.

To execute the seed-data.sh you need to have cURL installed on your local machine.

The seed-data.sh script runs the following actions:

  • Add 8 book categories.
  • Update 3 book categories.
  • Delete 2 book categories.
  • Add 17 books into the store.
  • Update 4 existing books.
  • Delete 2 existing books.
  • Add inventory for every book on the store.
  • Create 10 orders.
  • Cancel 3 existing orders.

The purpose behind this script is to generate a decent amount of business metrics that later can be visualized in Grafana and Prometheus.