We had four log formats. JSON-with-some-conventions, Logfmt-with-other- conventions, a homegrown thing one of us wrote in 2021, and the kernel ring buffer. Consolidating them onto OTLP took eight months and one intervention.
What worked
The OTLP wire format. It's verbose but it's predictable, and the SDK ecosystem actually exists.
What didn't
The migration plan. We wrote it assuming we could do the producers and the consumers in parallel. We couldn't. Producers always shipped first; consumers caught up over six weeks of follow-up work.