observable-api

This repo showcases observability concepts using a Spring Boot based application

Tech Stack

Concept	Tech
Language	Kotlin + JVM 17
REST	Spring Boot Reactive
DB	Postgresql + FlywayDB
Tracing	Opentracing + Jaeger
Metrics	Micrometer
Logging	Spring Boot default (logback) + Spring AOP

Observability concepts

Logs

A few best-practices have been implemented here to help various stakeholders

Prior to the spring boot server startup, we dump all active configuration properties in DEBUG. This can help developers or operations to confirm the application setup
We log in INFO general information/steps at key places to indicate flow progression
We log in INFO business rules traceability that can help QA/analysts/devs
We log in INFO audit entries: Who (user) queried what (endpoint) when (log timestamp)
We log in DEBUG using Spring AOP (Aspect-Oriented Programming = cross-cutting) to log the call flow

Traces

Note that opentracing is deprecated and effort is being put to evolve OpenTelemetry. For the purpose of this example, opentracing and jaeger is a mature integration while the spring opentelemetry project is currently in experimental phase. For more: Spring Experimental Projects

In the code, there's an example where we create a new child span (Span reported: e965ef21f008a949:d68675a8ac3fe2c2:e965ef21f008a949:1 - car-svc-getAllCars). The Span ID "112eec40fa03de27" is created to visualize a distinct traceability of the Service layer (CarService)

2022-04-01 14:39:33.828  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.audit.AuditRequest     : User [kevsuperduperuser] Queried /cars
2022-04-01 14:39:33.829  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.handler.CarsHandler    : About to query for all cars from the DB
2022-04-01 14:39:33.829 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.handler.CarsHandler    : cars() START - About to call CarService
2022-04-01 14:39:33.829 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     : getAllCars() START - About to call CarRepository
2022-04-01 14:39:33.928  INFO 18393 --- [or-http-epoll-2] i.j.internal.reporters.LoggingReporter   : Span reported: e965ef21f008a949:6c94ee4d37383003:e965ef21f008a949:1 - Query
2022-04-01 14:39:33.950  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Number of cars returned from DB 4
2022-04-01 14:39:33.950 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Cars entities returned:
2022-04-01 14:39:33.950 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Car = Car(id=1, name='Audi A4', msrp=44000, inventory=1000)
2022-04-01 14:39:33.954 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Car = Car(id=2, name='Audi S3', msrp=48000, inventory=2000)
2022-04-01 14:39:33.954 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Car = Car(id=3, name='Audi S4', msrp=61000, inventory=1800)
2022-04-01 14:39:33.954 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Car = Car(id=4, name='Audi RS6', msrp=125000, inventory=20)
2022-04-01 14:39:33.955 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     :    Obtained carsEntities. Prepare entities into DTO
2022-04-01 14:39:33.955 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.delivery.CarDelivery   : getPreparedData() START - Prepare entities into DTOs
2022-04-01 14:39:33.955  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.delivery.CarDelivery   : CD01 - Begin reviewing cars inventory
2022-04-01 14:39:33.955  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.delivery.CarDelivery   : CD01 - Reviewed
2022-04-01 14:39:33.955  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.delivery.CarDelivery   : CD02 - Begin adding supplementary info
2022-04-01 14:39:33.955  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.delivery.CarDelivery   : CD02 - Done adding supplementary info
2022-04-01 14:39:33.956 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.delivery.CarDelivery   : getPreparedData() END - Return carsDtos. Count = 4
2022-04-01 14:39:33.956 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.service.CarService     : getAllCars() END - Return DTOs
2022-04-01 14:39:33.956  INFO 18393 --- [or-http-epoll-2] i.j.internal.reporters.LoggingReporter   : Span reported: e965ef21f008a949:d68675a8ac3fe2c2:e965ef21f008a949:1 - car-svc-getAllCars
2022-04-01 14:39:33.956 DEBUG 18393 --- [or-http-epoll-2] c.k.observableapi.handler.CarsHandler    : cars() END - CarService called
2022-04-01 14:39:33.956  INFO 18393 --- [or-http-epoll-2] c.k.observableapi.handler.CarsHandler    : Done querying for all cars.
2022-04-01 14:39:34.017  INFO 18393 --- [or-http-epoll-2] i.j.internal.reporters.LoggingReporter   : Span reported: e965ef21f008a949:e965ef21f008a949:0:1 - /cars

Metrics

Prometheus-compatible metrics are implemented using micrometer and exposed using an actuator endpoint:

Page	URL
Actuator	http://localhost:8080/actuator/
Prometheus	http://localhost:8080/actuator/prometheus

In the class CarDelivery, we also implement a custom metric called cd02-delivery demonstrating custom metrics

After performing an API call of the GET endpoint, we can see this metric being available for Prometheus:

# HELP cd02_delivery_total The number of occurrences a request for inventory has successfully been enriched with delivery information
# TYPE cd02_delivery_total counter
cd02_delivery_total{businessrule="cd02",} 2.0

2 means that incremented twice in the CarDelivery class the enrichment of cars inventory. git When investigaitng API calls, this kind of business-logic metric can be used to cross reference network calls, to see if the counts are aligned such as here: 1 GET call = 1 increase of this cd02_delivery counter

Build and Deploy

Locally (using Docker)

1) Run Postgres Database

docker container run -d --name observableapidb -p 5432:5432 -e POSTGRES_PASSWORD=db1234 postgres:alpine

2) Run Jaeger

From https://www.jaegertracing.io/docs/1.33/getting-started/

docker container run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.33

Jaeger UI: http://localhost:16686/

3) Run the API

The API will pre-populate the database (DDL + DML) using flyway

./gradlew bootRun

on Kubernetes

1) Build the API using Maven Jib

./gradlew jibDockerBuild

TBD (k8s API manifests + steps)

API calls

Actuator

http://localhost:8080/actuator

Here, we can see results of our custom implementation of the Postgres Health Check

http://localhost:8080/actuator/health

API

Working (HTTP 200) GET Call

Pass the notion of a "user" using the header key "X-USER"

showcases logging for various stakeholders (audit/security, analysts, devs/ops)
showcases custom spans

curl -v -H "X-USER: kevsuperduperuser" localhost:8080/cars

After every call, see metrics: http://localhost:8080/actuator/prometheus

http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/cars",} 1.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/cars",} 0.131502888
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/cars",} 0.131502888

Metrics of interest here:

Metric	Description
http_server_requests_seconds_count	Sum of requests at this endpoint
http_server_requests_seconds_max	Sum of duration of every request at this endpoint

Non-Working (HTTP 5xx) POST call

curl -v -X POST -H "X-USER: kevsuperduperuser" localhost:8080/cars/reserve

showcases tracing visualisation
showcases metrics (http5xx)

After every call, see metrics: http://localhost:8080/actuator/prometheus

http_server_requests_seconds_count{exception="None",method="POST",outcome="SERVER_ERROR",status="500",uri="/cars/reserve",} 3.0
http_server_requests_seconds_sum{exception="None",method="POST",outcome="SERVER_ERROR",status="500",uri="/cars/reserve",} 0.003941968
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="POST",outcome="SERVER_ERROR",status="500",uri="/cars/reserve",} 0.001958607

Name		Name	Last commit message	Last commit date
Latest commit History 24 Commits
doc		doc
gradle/wrapper		gradle/wrapper
k8s		k8s
src		src
.gitignore		.gitignore
README.md		README.md
build.gradle.kts		build.gradle.kts
gradlew		gradlew
gradlew.bat		gradlew.bat
settings.gradle.kts		settings.gradle.kts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

observable-api

Tech Stack

Observability concepts

Logs

Traces

Metrics

Build and Deploy

Locally (using Docker)

1) Run Postgres Database

2) Run Jaeger

3) Run the API

on Kubernetes

1) Build the API using Maven Jib

API calls

Actuator

API

Working (HTTP 200) GET Call

Non-Working (HTTP 5xx) POST call

About

Uh oh!

Releases

Packages

Languages

kevvlvl/observable-api

Folders and files

Latest commit

History

Repository files navigation

observable-api

Tech Stack

Observability concepts

Logs

Traces

Metrics

Build and Deploy

Locally (using Docker)

1) Run Postgres Database

2) Run Jaeger

3) Run the API

on Kubernetes

1) Build the API using Maven Jib

API calls

Actuator

API

Working (HTTP 200) GET Call

Non-Working (HTTP 5xx) POST call

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages