Build a Span with a given SpanContext

[felixbarny]

Hi folks,

in Java terms I'd like to be be able to do the following:

SpanContext context = getSpanContext(request);
tracer.buildSpan("operationName")
    .withContext(context)
    .start();

-> Introduce SpanBuilder#withContext(SpanContext)

Right now, its only possible to create a child span given a SpanContext. I want to be able to create a span with a specific SpanContext.

The use case I have in mind is when a Span is created via JavaScript and then reported to a Java backend. In this backend, I want to recreate the span created in JavaScript, possibly enhance it with some tags (like adding the geo location based on the client ip, parsing of the user agent) and then report it. This is currently not possible in an implementation independent way.

Another use case might be ETL when you want to migrate traces from a legacy system to a new backend. In this case you probably also want to read traces from the old backend, transform them to a canonical format (Span) and then convert them to the target format.

[yurishkuro]

@felixbarny I can't seem to find that issue, but we discussed in the past a different span reference type like "decorates", i.e. if you have a span context for span A and you create span B with Reference(ctx=spanCtxForSpanA, refType="decorates"), then the backend may merge the tags/logs from B into A on ingestion or display (or use some sort of different visualization). In this approach we don't need to change the builder interface.

[felixbarny]

then the backend may merge the tags/logs from B into A on ingestion or display (or use some sort of different visualization)

Adding a reference does not seem to be quite the the semantics I'm aiming for. How is refType = decorates implemented in Jaeger Java?

If there are magic values for reference types, we should probably document them in https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans.

[yurishkuro]

Adding a reference does not seem to be quite the the semantics I'm aiming for.

Well, what seemed odd in your approach is that you're saying you have a Javascript layer that is somehow able to generate an appropriate SpanContext on the wire, which ideally should only be done with an OpenTracing tracer implementation. If you already have such implementation in Javascript, then it should be able to generate the Span in the respective backend format instead of relying on an extra Java server to do the same. The Java server may still want to enrich the span with additional tags, which is what the decorate reference allows.

How is refType = decorates implemented in Jaeger Java?

It's not yet, as I said it was just one of the suggestions on some github issue, which, sadly, I cannot find, perhaps it wasn't in the specification repo. @tedsuo do you remember?

If there are magic values for reference types, we should probably document

Definitely, once there's an agreement.

[felixbarny]

generate an appropriate SpanContext on the wire, which ideally should only be done with an OpenTracing tracer implementation

The way I see it every tracing system with uses the same approach for context propagation can create a span context.

In this instance, I'm using Instana's open source end user monitoring library Weasel.

If you already have such implementation in Javascript, then it should be able to generate the Span in the respective backend format instead of relying on an extra Java server to do the same.

Weasel does not create spans in the specific span format I need. In fact, that's also not desirable as you want a much more compressed format when sending beacons back to the server. A weasel beacon might look like this:

http://localhost:9966/petclinic/stagemonitor/public/eum?r=1500460599181&ts=2109&d=2115&ty=xhr&pl=848fad1487a95476&l=http%3A%2F%2Flocalhost%3A9966%2Fpetclinic%2F&m=GET&u=owners.html%3FlastName%3D&a=1&st=200&t=982dc17f92d3ee7e&s=982dc17f92d3ee7e

Where the parameters t and s stand for Trace-Id and Span-Id respectively.

Weasel sends this beacon after the ajax request has completed. It also modifies the XMLHttpRequest so that the B3 Headers are propagated.

The Java server may still want to enrich the span with additional tags, which is what the decorate reference allows.

The Java server does not decorate an existing span but rather creates a span from a beacon with a given span context (t and s parameters).

---

I don't think the general principle is specific to my stagemonitor-weasel beacon-span-creating use case. As I mentioned it's just one example of a much broader use case called ETL.

[yurishkuro]

@felixbarny note that there is a significant difference in creating a new span that decorates another span, and re-creating another instance of the span with the existing span ID. The latter may not even be supported by some tracing systems as it means multiple spans with the same ID.

[mikewrighton]

FWIW I also have a use case for the 'decoration' of existing spans. I'm trying to add log fields from a Log4j-based logging system, but the log appender runs asynchronously meaning that by the time the log entry is processed, the span which it corresponds to has already been sent out. Being able to add additional log fields/tags for a known span ID would solve this problem.

[felixbarny]

@felixbarny note that there is a significant difference in creating a new span that decorates another span, and re-creating another instance of the span with the existing span ID.

I agree.

The latter may not even be supported by some tracing systems as it means multiple spans with the same ID.

This issue is about adding support for that on the API level. Or do you mean it is technically impossible for some tracer implementations? If so, why?

the log appender runs asynchronously meaning that by the time the log entry is processed, the span which it corresponds to has already been sent out

You are trying to implement a log appender which adds logs to the active span? Interesting! Have you tried using a non-async logger for that? While a very interesting one, I don't think this is the use case I was trying to describe in this issue.

[mikewrighton]

You are trying to implement a log appender which adds logs to the active span? Interesting! Have you tried using a non-async logger for that?

Yep - 'active' as in active at the time the log method was called by the user. Really I'm just trying to replicate the zap functionality but with Log4j. A synchronous log appender would work but is not really an option for performance reasons.

[felixbarny]

It would just be adding a log entry to an in memory structure. No need to go async here. Your ot impl of choice will most likely report the logs with the span once its finished asynchronously. mikewrighton <notifications@github.com> schrieb am Fr., 28. Juli 2017 17:49:

…

You are trying to implement a log appender which adds logs to the active span? Interesting! Have you tried using a non-async logger for that? Yep - 'active' as in active at the time the log method was called by the user. A synchronous log appender would work but is not really an option for performance reasons. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACEDCOOmiYI5S_EVXq18nXv-XrD-Alf0ks5sSgMBgaJpZM4OaEdf> .

[mikewrighton]

@felixbarny Yeah it's a good point - I wasn't sure how much log processing would be in the critical path e.g. variable processing, but I think it looks fine, so I'll go ahead with a synchronous appender. Thanks for the help!

[yurishkuro]

This issue is about adding support for that on the API level. Or do you mean it is technically impossible for some tracer implementations? If so, why?

@felixbarny I cannot answer the question about all implementations, that's why adding something like that to the API should be discussed more broadly.

I think the "span enrichment" use case that @mikewrighton has is easier to solve, with a new span reference type. The ETL scenario is not a very clean use case in the first place, because the assumption is that you want to transform something that is not produced by an OpenTracing implementation into a Span while remaining vendor-agnostic in the transformation. But how can you be vendor neutral if you already depend on the specific format of the SpanContext? You are making the assumption that your JS source generates the SpanContext in the exact same way as the backend tracer, which implies tight vendor coupling. Why not then just create the Span manually using vendor-specific API, like new com.uber.jaerger.Span(...)?

I am not trying to dismiss your use case, I think it is indeed an important one to support, but I can see it getting easier to support once we are able to standardize on things like SpanContext wire representation (e.g. https://github.com/TraceContext/tracecontext-spec), and possibly vendor-neutral Span representation (e.g. #64).

[felixbarny]

that's why adding something like that to the API should be discussed more broadly.

Before talking about if its possible to implement we should agree if it makes sense to add the method. What is you opinion on that?

I cannot answer the question about all implementations

What about Jaeger, would it be possible to add it there?

But how can you be vendor neutral if you already depend on the specific format of the SpanContext? You are making the assumption that your JS source generates the SpanContext in the exact same way as the backend tracer, which implies tight vendor coupling.

The only assumption I make is that the source (JavaScript) produces a B3 compatible SpanContext. I don't see a vendor coupling here.

Why not then just create the Span manually using vendor-specific API, like new com.uber.jaerger.Span(...)?

Because I want to be able to use different OT Impls.

[yurishkuro]

The only assumption I make is that the source (JavaScript) produces a B3 compatible SpanContext.

But in order to do .withContext(context) you still need to create a vendor-specific SpanContext first.

Before talking about if its possible to implement we should agree if it makes sense to add the method. What is you opinion on that?

In theory it makes sense to me, but I think the standard for wire format is a pre-requisite, otherwise I still don't see how you can make your code completely vendor neutral.

What about Jaeger, would it be possible to add it there?

The simple ETL case wouldn't break Jaeger, because it still means each span will have a unique span ID. The enrichment case other people asked for, such as adding tags/logs to the existing span via another instance with the same span ID, will probably cause an issue. Jaeger backend does allow multiple spans with the same ID, but that was only done to support Zipkin's shared RPC span model and those client-server spans are actually deduped before returning to the UI. Additionally, we had an internal process where we parse haproxy logs and store them by writing another span with the same ID. To deal with that the resulting dups we have a pre-processor in the query service that finds those haproxy spans and merges them into the real span by essentially discarding everything except logs. A general solution for multiple spans with the same ID requires certain merging rules, like in Zipkin where it does min(startTime), max(duration), union(logs), and then somewhat funky logic about deciding the operation name (certain precedence order). By going with a decorator reference type we avoid the need for the merging.

[felixbarny]

But in order to do .withContext(context) you still need to create a vendor-specific SpanContext first.

Yes, but it is independent form the OT Impl: I can create the context with Tracer#extract and a custom Format (aka extractor) which reads the weasel specific URL parameters from the HTTP request.

Sure enough, I have to write an extractor which is specific to Weasel but it is agnostic of the underlying OT-Impl.

[yurishkuro]

Sure enough, I have to write an extractor which is specific to Weasel but it is agnostic of the underlying OT-Impl.

Yes, assuming that you only deal with OT impls that understand a common wire format like B3