gRPC server instrumentation: no spans created for unknown service

[lmolkova]

Let's say I made a configuration mistake: my gRPC client calls serivce-a, but it's not registered on the gRPC server I called.

Expected: grpc instrumentation records a client and server spans, both have error code set to UNIMPLEMENTED.
Actual: only client span is recorded, server span is not recorded at all

It happens because interceptors are not called if service is not found.
Interestingly enough native gRPC instrumentation does trace the span. It uses ServerStreamTracer.Factory mechanism instead of interceptors.

It may be configured similarly to interceptors

server = ServerBuilder.forPort(port)
        .intercept(new ServerInterceptor() {
            @Override
            public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(ServerCall<ReqT, RespT> call, Metadata headers, ServerCallHandler<ReqT, RespT> next) {
                // this is what we use today and it's not called with unknown services
                return next.startCall(call, headers);
            }
        })
        .addService(new GreeterImpl())
        .addStreamTracerFactory(new ServerStreamTracer.Factory() {

            @Override
            public ServerStreamTracer newServerStreamTracer(String fullMethodName, Metadata headers) {
                return new ServerStreamTracer() {
                      // this one is used by native grpc instrumentation and is called
                      ...
                };
            }
        })
        .build();

Note: tracing unknown methods leads to potential unbound cardinality around service and methods name.
See open-telemetry/semantic-conventions#3196 (comment) for the details.

The proposed (in open-telemetry/semantic-conventions#3223) solutions would be to:

• use ServerStreamTracer to start span, delay setting rpc.service | method attribute(s), initial span name would be _OTHER
• use server interceptor as well - when interceptor is called, it means method is available on the server update span name to actual method name
• populate rpc.method | service attribute(s) when span ends
• support config option to set 'known methods'.
  • if it's not provided, use above logic, method name then is not available for sampling
  • If it's provided, span name and method name would be populated at start time according to the config and available for sampling. No need for logic in the interceptor.
  • this config option also helps with edge case when server accepts arbitrary methods.