[{"data":1,"prerenderedAt":1774},["ShallowReactive",2],{"blog-en-opentelemetry-distributed-tracing":3,"blog-en-opentelemetry-distributed-tracing-alt":531},{"id":4,"title":5,"author":6,"body":7,"date":1758,"description":1759,"extension":1760,"image":95,"locale":1761,"meta":1762,"navigation":531,"path":1763,"seo":1764,"stem":1765,"tags":1766,"__hash__":1773},"blog\u002Fblog\u002Fen\u002Fopentelemetry-distributed-tracing.md","Implementing Distributed Tracing with OpenTelemetry: From Basics to Production","Kubo Team",{"type":8,"value":9,"toc":1731},"minimark",[10,28,33,38,47,51,58,81,85,96,116,124,128,143,147,153,157,425,429,735,742,746,750,757,762,796,889,894,937,941,950,1036,1040,1149,1156,1160,1164,1177,1278,1282,1291,1412,1416,1428,1434,1437,1441,1445,1454,1628,1632,1658,1667,1671,1674,1706,1715,1727],[11,12,13,14,21,22,27],"p",{},"In a microservices architecture, a single request traverses multiple services. When problems occur, pinpointing which service and which operation is causing latency is far from trivial. ",[15,16,20],"a",{"href":17,"rel":18},"https:\u002F\u002Fopentelemetry.io\u002F",[19],"nofollow","OpenTelemetry",", a CNCF Graduated project, standardizes distributed tracing and solves this challenge as a comprehensive observability framework. Even on lightweight K3s-based Kubernetes environments like ",[15,23,26],{"href":24,"rel":25},"https:\u002F\u002Fkubo.hexabase.io\u002F",[19],"Kubo",", OpenTelemetry provides complete visibility into inter-service request flows.",[29,30,32],"h2",{"id":31},"distributed-tracing-fundamentals","Distributed Tracing Fundamentals",[34,35,37],"h3",{"id":36},"why-distributed-tracing-matters","Why Distributed Tracing Matters",[11,39,40,41,46],{},"As ",[15,42,45],{"href":43,"rel":44},"https:\u002F\u002Fthenewstack.io\u002Fhow-opentelemetry-works-tracing-metrics-and-logs-on-kubernetes\u002F",[19],"The New Stack"," points out, metrics and logs alone are insufficient in Kubernetes microservice environments. To understand the complete picture as a request crosses pods, nodes, and namespaces, distributed tracing is essential.",[34,48,50],{"id":49},"opentelemetry-telemetry-signals","OpenTelemetry Telemetry Signals",[11,52,53,57],{},[15,54,20],{"href":55,"rel":56},"https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002F",[19]," is a vendor-neutral observability framework that unifies three telemetry signals:",[59,60,61,69,75],"ul",{},[62,63,64,68],"li",{},[65,66,67],"strong",{},"Traces",": Track the end-to-end flow of requests. Parent-child relationships between Spans represent service call chains",[62,70,71,74],{},[65,72,73],{},"Metrics",": Quantitative measurements such as request counts, latency, and error rates",[62,76,77,80],{},[65,78,79],{},"Logs",": Event records that, when correlated with trace IDs, identify logs related to specific requests",[34,82,84],{"id":83},"tracing-building-blocks","Tracing Building Blocks",[86,87,92],"pre",{"className":88,"code":90,"language":91},[89],"language-text","Trace\n├── Span A (API Gateway, 150ms)\n│   ├── Span B (Auth Service, 20ms)\n│   └── Span C (Product Service, 100ms)\n│       ├── Span D (Database Query, 40ms)\n│       └── Span E (Cache Lookup, 5ms)\n","text",[93,94,90],"code",{"__ignoreMap":95},"",[59,97,98,104,110],{},[62,99,100,103],{},[65,101,102],{},"Trace",": The overall processing flow of a single request",[62,105,106,109],{},[65,107,108],{},"Span",": An individual unit of work within a Trace, with start time, end time, attributes, and events",[62,111,112,115],{},[65,113,114],{},"Context",": Information containing Trace ID and Span ID, propagated between services",[11,117,118,123],{},[15,119,122],{"href":120,"rel":121},"https:\u002F\u002Fwww.hexabase.com\u002Fproduct\u002Fcaptain-ai\u002F",[19],"Captain.AI"," uses AI to analyze tracing data, automatically detecting performance bottlenecks and suggesting optimizations.",[29,125,127],{"id":126},"configuring-the-opentelemetry-collector","Configuring the OpenTelemetry Collector",[11,129,130,131,136,137,142],{},"The OpenTelemetry Collector is a vendor-agnostic component responsible for receiving, processing, and exporting telemetry data. This section draws from ",[15,132,135],{"href":133,"rel":134},"https:\u002F\u002Fuptrace.dev\u002Fopentelemetry\u002Fdistributed-tracing",[19],"Uptrace"," and the ",[15,138,141],{"href":139,"rel":140},"https:\u002F\u002Flogit.io\u002Fblog\u002Fpost\u002Fopentelemetry-distributed-tracing-implementation\u002F",[19],"Logit.io implementation guide",".",[34,144,146],{"id":145},"collector-architecture","Collector Architecture",[86,148,151],{"className":149,"code":150,"language":91},[89],"Receivers → Processors → Exporters\n  (OTLP)     (batch)      (Jaeger)\n  (Zipkin)   (filter)     (Tempo)\n  (Jaeger)   (sampling)   (OTLP)\n",[93,152,150],{"__ignoreMap":95},[34,154,156],{"id":155},"kubernetes-deployment-daemonset","Kubernetes Deployment (DaemonSet)",[86,158,162],{"className":159,"code":160,"language":161,"meta":95,"style":95},"language-yaml shiki shiki-themes tokyo-night","apiVersion: apps\u002Fv1\nkind: DaemonSet\nmetadata:\n  name: otel-collector\n  namespace: observability\nspec:\n  selector:\n    matchLabels:\n      app: otel-collector\n  template:\n    metadata:\n      labels:\n        app: otel-collector\n    spec:\n      containers:\n      - name: collector\n        image: otel\u002Fopentelemetry-collector-contrib:0.98.0\n        args: [\"--config=\u002Fconf\u002Fotel-collector-config.yaml\"]\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fconf\n      volumes:\n      - name: config\n        configMap:\n          name: otel-collector-config\n","yaml",[93,163,164,181,192,201,212,223,231,239,247,257,265,273,281,291,299,307,322,333,355,363,376,387,395,406,414],{"__ignoreMap":95},[165,166,169,173,177],"span",{"class":167,"line":168},"line",1,[165,170,172],{"class":171},"s0U2E","apiVersion",[165,174,176],{"class":175},"sAklC",":",[165,178,180],{"class":179},"sPY7s"," apps\u002Fv1\n",[165,182,184,187,189],{"class":167,"line":183},2,[165,185,186],{"class":171},"kind",[165,188,176],{"class":175},[165,190,191],{"class":179}," DaemonSet\n",[165,193,195,198],{"class":167,"line":194},3,[165,196,197],{"class":171},"metadata",[165,199,200],{"class":175},":\n",[165,202,204,207,209],{"class":167,"line":203},4,[165,205,206],{"class":171},"  name",[165,208,176],{"class":175},[165,210,211],{"class":179}," otel-collector\n",[165,213,215,218,220],{"class":167,"line":214},5,[165,216,217],{"class":171},"  namespace",[165,219,176],{"class":175},[165,221,222],{"class":179}," observability\n",[165,224,226,229],{"class":167,"line":225},6,[165,227,228],{"class":171},"spec",[165,230,200],{"class":175},[165,232,234,237],{"class":167,"line":233},7,[165,235,236],{"class":171},"  selector",[165,238,200],{"class":175},[165,240,242,245],{"class":167,"line":241},8,[165,243,244],{"class":171},"    matchLabels",[165,246,200],{"class":175},[165,248,250,253,255],{"class":167,"line":249},9,[165,251,252],{"class":171},"      app",[165,254,176],{"class":175},[165,256,211],{"class":179},[165,258,260,263],{"class":167,"line":259},10,[165,261,262],{"class":171},"  template",[165,264,200],{"class":175},[165,266,268,271],{"class":167,"line":267},11,[165,269,270],{"class":171},"    metadata",[165,272,200],{"class":175},[165,274,276,279],{"class":167,"line":275},12,[165,277,278],{"class":171},"      labels",[165,280,200],{"class":175},[165,282,284,287,289],{"class":167,"line":283},13,[165,285,286],{"class":171},"        app",[165,288,176],{"class":175},[165,290,211],{"class":179},[165,292,294,297],{"class":167,"line":293},14,[165,295,296],{"class":171},"    spec",[165,298,200],{"class":175},[165,300,302,305],{"class":167,"line":301},15,[165,303,304],{"class":171},"      containers",[165,306,200],{"class":175},[165,308,310,314,317,319],{"class":167,"line":309},16,[165,311,313],{"class":312},"sgJMe","      -",[165,315,316],{"class":171}," name",[165,318,176],{"class":175},[165,320,321],{"class":179}," collector\n",[165,323,325,328,330],{"class":167,"line":324},17,[165,326,327],{"class":171},"        image",[165,329,176],{"class":175},[165,331,332],{"class":179}," otel\u002Fopentelemetry-collector-contrib:0.98.0\n",[165,334,336,339,341,344,347,350,352],{"class":167,"line":335},18,[165,337,338],{"class":171},"        args",[165,340,176],{"class":175},[165,342,343],{"class":175}," [",[165,345,346],{"class":175},"\"",[165,348,349],{"class":179},"--config=\u002Fconf\u002Fotel-collector-config.yaml",[165,351,346],{"class":175},[165,353,354],{"class":175},"]\n",[165,356,358,361],{"class":167,"line":357},19,[165,359,360],{"class":171},"        volumeMounts",[165,362,200],{"class":175},[165,364,366,369,371,373],{"class":167,"line":365},20,[165,367,368],{"class":312},"        -",[165,370,316],{"class":171},[165,372,176],{"class":175},[165,374,375],{"class":179}," config\n",[165,377,379,382,384],{"class":167,"line":378},21,[165,380,381],{"class":171},"          mountPath",[165,383,176],{"class":175},[165,385,386],{"class":179}," \u002Fconf\n",[165,388,390,393],{"class":167,"line":389},22,[165,391,392],{"class":171},"      volumes",[165,394,200],{"class":175},[165,396,398,400,402,404],{"class":167,"line":397},23,[165,399,313],{"class":312},[165,401,316],{"class":171},[165,403,176],{"class":175},[165,405,375],{"class":179},[165,407,409,412],{"class":167,"line":408},24,[165,410,411],{"class":171},"        configMap",[165,413,200],{"class":175},[165,415,417,420,422],{"class":167,"line":416},25,[165,418,419],{"class":171},"          name",[165,421,176],{"class":175},[165,423,424],{"class":179}," otel-collector-config\n",[34,426,428],{"id":427},"collector-configuration","Collector Configuration",[86,430,432],{"className":159,"code":431,"language":161,"meta":95,"style":95},"apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: otel-collector-config\n  namespace: observability\ndata:\n  otel-collector-config.yaml: |\n    receivers:\n      otlp:\n        protocols:\n          grpc:\n            endpoint: 0.0.0.0:4317\n          http:\n            endpoint: 0.0.0.0:4318\n\n    processors:\n      batch:\n        timeout: 5s\n        send_batch_size: 1024\n      memory_limiter:\n        check_interval: 1s\n        limit_mib: 512\n      tail_sampling:\n        decision_wait: 10s\n        policies:\n        - name: errors-policy\n          type: status_code\n          status_code: {status_codes: [ERROR]}\n        - name: slow-traces\n          type: latency\n          latency: {threshold_ms: 1000}\n        - name: probabilistic\n          type: probabilistic\n          probabilistic: {sampling_percentage: 10}\n\n    exporters:\n      otlp\u002Ftempo:\n        endpoint: tempo:4317\n        tls:\n          insecure: true\n      otlp\u002Fjaeger:\n        endpoint: jaeger-collector:4317\n        tls:\n          insecure: true\n\n    service:\n      pipelines:\n        traces:\n          receivers: [otlp]\n          processors: [memory_limiter, tail_sampling, batch]\n          exporters: [otlp\u002Ftempo]\n",[93,433,434,443,452,458,466,474,481,492,497,502,507,512,517,522,527,533,538,543,548,553,558,563,568,573,578,583,589,595,601,607,613,619,625,631,637,642,648,654,660,666,672,678,684,689,694,699,705,711,717,723,729],{"__ignoreMap":95},[165,435,436,438,440],{"class":167,"line":168},[165,437,172],{"class":171},[165,439,176],{"class":175},[165,441,442],{"class":179}," v1\n",[165,444,445,447,449],{"class":167,"line":183},[165,446,186],{"class":171},[165,448,176],{"class":175},[165,450,451],{"class":179}," ConfigMap\n",[165,453,454,456],{"class":167,"line":194},[165,455,197],{"class":171},[165,457,200],{"class":175},[165,459,460,462,464],{"class":167,"line":203},[165,461,206],{"class":171},[165,463,176],{"class":175},[165,465,424],{"class":179},[165,467,468,470,472],{"class":167,"line":214},[165,469,217],{"class":171},[165,471,176],{"class":175},[165,473,222],{"class":179},[165,475,476,479],{"class":167,"line":225},[165,477,478],{"class":171},"data",[165,480,200],{"class":175},[165,482,483,486,488],{"class":167,"line":233},[165,484,485],{"class":171},"  otel-collector-config.yaml",[165,487,176],{"class":175},[165,489,491],{"class":490},"sd1Qi"," |\n",[165,493,494],{"class":167,"line":241},[165,495,496],{"class":179},"    receivers:\n",[165,498,499],{"class":167,"line":249},[165,500,501],{"class":179},"      otlp:\n",[165,503,504],{"class":167,"line":259},[165,505,506],{"class":179},"        protocols:\n",[165,508,509],{"class":167,"line":267},[165,510,511],{"class":179},"          grpc:\n",[165,513,514],{"class":167,"line":275},[165,515,516],{"class":179},"            endpoint: 0.0.0.0:4317\n",[165,518,519],{"class":167,"line":283},[165,520,521],{"class":179},"          http:\n",[165,523,524],{"class":167,"line":293},[165,525,526],{"class":179},"            endpoint: 0.0.0.0:4318\n",[165,528,529],{"class":167,"line":301},[165,530,532],{"emptyLinePlaceholder":531},true,"\n",[165,534,535],{"class":167,"line":309},[165,536,537],{"class":179},"    processors:\n",[165,539,540],{"class":167,"line":324},[165,541,542],{"class":179},"      batch:\n",[165,544,545],{"class":167,"line":335},[165,546,547],{"class":179},"        timeout: 5s\n",[165,549,550],{"class":167,"line":357},[165,551,552],{"class":179},"        send_batch_size: 1024\n",[165,554,555],{"class":167,"line":365},[165,556,557],{"class":179},"      memory_limiter:\n",[165,559,560],{"class":167,"line":378},[165,561,562],{"class":179},"        check_interval: 1s\n",[165,564,565],{"class":167,"line":389},[165,566,567],{"class":179},"        limit_mib: 512\n",[165,569,570],{"class":167,"line":397},[165,571,572],{"class":179},"      tail_sampling:\n",[165,574,575],{"class":167,"line":408},[165,576,577],{"class":179},"        decision_wait: 10s\n",[165,579,580],{"class":167,"line":416},[165,581,582],{"class":179},"        policies:\n",[165,584,586],{"class":167,"line":585},26,[165,587,588],{"class":179},"        - name: errors-policy\n",[165,590,592],{"class":167,"line":591},27,[165,593,594],{"class":179},"          type: status_code\n",[165,596,598],{"class":167,"line":597},28,[165,599,600],{"class":179},"          status_code: {status_codes: [ERROR]}\n",[165,602,604],{"class":167,"line":603},29,[165,605,606],{"class":179},"        - name: slow-traces\n",[165,608,610],{"class":167,"line":609},30,[165,611,612],{"class":179},"          type: latency\n",[165,614,616],{"class":167,"line":615},31,[165,617,618],{"class":179},"          latency: {threshold_ms: 1000}\n",[165,620,622],{"class":167,"line":621},32,[165,623,624],{"class":179},"        - name: probabilistic\n",[165,626,628],{"class":167,"line":627},33,[165,629,630],{"class":179},"          type: probabilistic\n",[165,632,634],{"class":167,"line":633},34,[165,635,636],{"class":179},"          probabilistic: {sampling_percentage: 10}\n",[165,638,640],{"class":167,"line":639},35,[165,641,532],{"emptyLinePlaceholder":531},[165,643,645],{"class":167,"line":644},36,[165,646,647],{"class":179},"    exporters:\n",[165,649,651],{"class":167,"line":650},37,[165,652,653],{"class":179},"      otlp\u002Ftempo:\n",[165,655,657],{"class":167,"line":656},38,[165,658,659],{"class":179},"        endpoint: tempo:4317\n",[165,661,663],{"class":167,"line":662},39,[165,664,665],{"class":179},"        tls:\n",[165,667,669],{"class":167,"line":668},40,[165,670,671],{"class":179},"          insecure: true\n",[165,673,675],{"class":167,"line":674},41,[165,676,677],{"class":179},"      otlp\u002Fjaeger:\n",[165,679,681],{"class":167,"line":680},42,[165,682,683],{"class":179},"        endpoint: jaeger-collector:4317\n",[165,685,687],{"class":167,"line":686},43,[165,688,665],{"class":179},[165,690,692],{"class":167,"line":691},44,[165,693,671],{"class":179},[165,695,697],{"class":167,"line":696},45,[165,698,532],{"emptyLinePlaceholder":531},[165,700,702],{"class":167,"line":701},46,[165,703,704],{"class":179},"    service:\n",[165,706,708],{"class":167,"line":707},47,[165,709,710],{"class":179},"      pipelines:\n",[165,712,714],{"class":167,"line":713},48,[165,715,716],{"class":179},"        traces:\n",[165,718,720],{"class":167,"line":719},49,[165,721,722],{"class":179},"          receivers: [otlp]\n",[165,724,726],{"class":167,"line":725},50,[165,727,728],{"class":179},"          processors: [memory_limiter, tail_sampling, batch]\n",[165,730,732],{"class":167,"line":731},51,[165,733,734],{"class":179},"          exporters: [otlp\u002Ftempo]\n",[11,736,737,738,741],{},"For K3s clusters running on ",[15,739,26],{"href":24,"rel":740},[19],", the DaemonSet approach is recommended -- one Collector per node maximizes resource efficiency.",[29,743,745],{"id":744},"application-instrumentation","Application Instrumentation",[34,747,749],{"id":748},"zero-code-instrumentation","Zero-Code Instrumentation",[11,751,752,756],{},[15,753,755],{"href":55,"rel":754},[19],"OpenTelemetry's"," auto-instrumentation allows you to add tracing without modifying code.",[11,758,759],{},[65,760,761],{},"Python example:",[86,763,767],{"className":764,"code":765,"language":766,"meta":95,"style":95},"language-bash shiki shiki-themes tokyo-night","pip install opentelemetry-distro opentelemetry-exporter-otlp\nopentelemetry-bootstrap -a install\n","bash",[93,768,769,784],{"__ignoreMap":95},[165,770,771,775,778,781],{"class":167,"line":168},[165,772,774],{"class":773},"sE3pS","pip",[165,776,777],{"class":179}," install",[165,779,780],{"class":179}," opentelemetry-distro",[165,782,783],{"class":179}," opentelemetry-exporter-otlp\n",[165,785,786,789,793],{"class":167,"line":183},[165,787,788],{"class":773},"opentelemetry-bootstrap",[165,790,792],{"class":791},"sT800"," -a",[165,794,795],{"class":179}," install\n",[86,797,799],{"className":764,"code":798,"language":766,"meta":95,"style":95},"# Configuration via environment variables\nexport OTEL_SERVICE_NAME=my-python-service\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http:\u002F\u002Fotel-collector:4317\nexport OTEL_TRACES_EXPORTER=otlp\nexport OTEL_METRICS_EXPORTER=otlp\n\n# Launch with auto-instrumentation\nopentelemetry-instrument python app.py\n",[93,800,801,807,822,846,858,869,873,878],{"__ignoreMap":95},[165,802,803],{"class":167,"line":168},[165,804,806],{"class":805},"sbD-w","# Configuration via environment variables\n",[165,808,809,813,816,819],{"class":167,"line":183},[165,810,812],{"class":811},"sN7LL","export",[165,814,815],{"class":773}," OTEL_SERVICE_NAME",[165,817,818],{"class":175},"=",[165,820,821],{"class":773},"my-python-service\n",[165,823,824,826,829,831,834,838,841,843],{"class":167,"line":194},[165,825,812],{"class":811},[165,827,828],{"class":773}," OTEL_EXPORTER_OTLP_ENDPOINT",[165,830,818],{"class":175},[165,832,833],{"class":773},"http",[165,835,837],{"class":836},"sGX4V",":\u002F\u002F",[165,839,840],{"class":773},"otel-collector",[165,842,176],{"class":836},[165,844,845],{"class":773},"4317\n",[165,847,848,850,853,855],{"class":167,"line":203},[165,849,812],{"class":811},[165,851,852],{"class":773}," OTEL_TRACES_EXPORTER",[165,854,818],{"class":175},[165,856,857],{"class":773},"otlp\n",[165,859,860,862,865,867],{"class":167,"line":214},[165,861,812],{"class":811},[165,863,864],{"class":773}," OTEL_METRICS_EXPORTER",[165,866,818],{"class":175},[165,868,857],{"class":773},[165,870,871],{"class":167,"line":225},[165,872,532],{"emptyLinePlaceholder":531},[165,874,875],{"class":167,"line":233},[165,876,877],{"class":805},"# Launch with auto-instrumentation\n",[165,879,880,883,886],{"class":167,"line":241},[165,881,882],{"class":773},"opentelemetry-instrument",[165,884,885],{"class":179}," python",[165,887,888],{"class":179}," app.py\n",[11,890,891],{},[65,892,893],{},"Java example:",[86,895,897],{"className":764,"code":896,"language":766,"meta":95,"style":95},"# Auto-instrumentation with Java Agent\njava -javaagent:opentelemetry-javaagent.jar \\\n  -Dotel.service.name=my-java-service \\\n  -Dotel.exporter.otlp.endpoint=http:\u002F\u002Fotel-collector:4317 \\\n  -jar my-app.jar\n",[93,898,899,904,915,922,929],{"__ignoreMap":95},[165,900,901],{"class":167,"line":168},[165,902,903],{"class":805},"# Auto-instrumentation with Java Agent\n",[165,905,906,909,912],{"class":167,"line":183},[165,907,908],{"class":773},"java",[165,910,911],{"class":791}," -javaagent:opentelemetry-javaagent.jar",[165,913,914],{"class":175}," \\\n",[165,916,917,920],{"class":167,"line":194},[165,918,919],{"class":791},"  -Dotel.service.name=my-java-service",[165,921,914],{"class":175},[165,923,924,927],{"class":167,"line":203},[165,925,926],{"class":791},"  -Dotel.exporter.otlp.endpoint=http:\u002F\u002Fotel-collector:4317",[165,928,914],{"class":175},[165,930,931,934],{"class":167,"line":214},[165,932,933],{"class":791},"  -jar",[165,935,936],{"class":179}," my-app.jar\n",[34,938,940],{"id":939},"automatic-injection-with-kubernetes-operator","Automatic Injection with Kubernetes Operator",[11,942,943,944,949],{},"As explained in this ",[15,945,948],{"href":946,"rel":947},"https:\u002F\u002Fmedium.com\u002F@serverwalainfra\u002Fimplementing-end-to-end-tracing-with-opentelemetry-in-kubernetes-workloads-e9ffbf5dc002",[19],"Medium implementation article",", the OpenTelemetry Operator enables auto-instrumentation injection via Pod annotations:",[86,951,953],{"className":159,"code":952,"language":161,"meta":95,"style":95},"apiVersion: v1\nkind: Pod\nmetadata:\n  annotations:\n    instrumentation.opentelemetry.io\u002Finject-python: \"true\"\nspec:\n  containers:\n  - name: my-app\n    image: my-python-app:latest\n",[93,954,955,963,972,978,985,1001,1007,1014,1026],{"__ignoreMap":95},[165,956,957,959,961],{"class":167,"line":168},[165,958,172],{"class":171},[165,960,176],{"class":175},[165,962,442],{"class":179},[165,964,965,967,969],{"class":167,"line":183},[165,966,186],{"class":171},[165,968,176],{"class":175},[165,970,971],{"class":179}," Pod\n",[165,973,974,976],{"class":167,"line":194},[165,975,197],{"class":171},[165,977,200],{"class":175},[165,979,980,983],{"class":167,"line":203},[165,981,982],{"class":171},"  annotations",[165,984,200],{"class":175},[165,986,987,990,992,995,998],{"class":167,"line":214},[165,988,989],{"class":171},"    instrumentation.opentelemetry.io\u002Finject-python",[165,991,176],{"class":175},[165,993,994],{"class":175}," \"",[165,996,997],{"class":179},"true",[165,999,1000],{"class":175},"\"\n",[165,1002,1003,1005],{"class":167,"line":225},[165,1004,228],{"class":171},[165,1006,200],{"class":175},[165,1008,1009,1012],{"class":167,"line":233},[165,1010,1011],{"class":171},"  containers",[165,1013,200],{"class":175},[165,1015,1016,1019,1021,1023],{"class":167,"line":241},[165,1017,1018],{"class":312},"  -",[165,1020,316],{"class":171},[165,1022,176],{"class":175},[165,1024,1025],{"class":179}," my-app\n",[165,1027,1028,1031,1033],{"class":167,"line":249},[165,1029,1030],{"class":171},"    image",[165,1032,176],{"class":175},[165,1034,1035],{"class":179}," my-python-app:latest\n",[34,1037,1039],{"id":1038},"manual-instrumentation-example-go","Manual Instrumentation Example (Go)",[86,1041,1045],{"className":1042,"code":1043,"language":1044,"meta":95,"style":95},"language-go shiki shiki-themes tokyo-night","import (\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n)\n\nfunc handleRequest(ctx context.Context) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"handleRequest\")\n    defer span.End()\n\n    \u002F\u002F Add custom attributes\n    span.SetAttributes(\n        attribute.String(\"user.id\", userID),\n        attribute.Int(\"http.status_code\", 200),\n    )\n\n    \u002F\u002F Create child span\n    ctx, childSpan := tracer.Start(ctx, \"database-query\")\n    result := queryDatabase(ctx)\n    childSpan.End()\n}\n","go",[93,1046,1047,1052,1057,1062,1067,1071,1076,1081,1086,1091,1095,1100,1105,1110,1115,1120,1124,1129,1134,1139,1144],{"__ignoreMap":95},[165,1048,1049],{"class":167,"line":168},[165,1050,1051],{},"import (\n",[165,1053,1054],{"class":167,"line":183},[165,1055,1056],{},"    \"go.opentelemetry.io\u002Fotel\"\n",[165,1058,1059],{"class":167,"line":194},[165,1060,1061],{},"    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n",[165,1063,1064],{"class":167,"line":203},[165,1065,1066],{},")\n",[165,1068,1069],{"class":167,"line":214},[165,1070,532],{"emptyLinePlaceholder":531},[165,1072,1073],{"class":167,"line":225},[165,1074,1075],{},"func handleRequest(ctx context.Context) {\n",[165,1077,1078],{"class":167,"line":233},[165,1079,1080],{},"    tracer := otel.Tracer(\"my-service\")\n",[165,1082,1083],{"class":167,"line":241},[165,1084,1085],{},"    ctx, span := tracer.Start(ctx, \"handleRequest\")\n",[165,1087,1088],{"class":167,"line":249},[165,1089,1090],{},"    defer span.End()\n",[165,1092,1093],{"class":167,"line":259},[165,1094,532],{"emptyLinePlaceholder":531},[165,1096,1097],{"class":167,"line":267},[165,1098,1099],{},"    \u002F\u002F Add custom attributes\n",[165,1101,1102],{"class":167,"line":275},[165,1103,1104],{},"    span.SetAttributes(\n",[165,1106,1107],{"class":167,"line":283},[165,1108,1109],{},"        attribute.String(\"user.id\", userID),\n",[165,1111,1112],{"class":167,"line":293},[165,1113,1114],{},"        attribute.Int(\"http.status_code\", 200),\n",[165,1116,1117],{"class":167,"line":301},[165,1118,1119],{},"    )\n",[165,1121,1122],{"class":167,"line":309},[165,1123,532],{"emptyLinePlaceholder":531},[165,1125,1126],{"class":167,"line":324},[165,1127,1128],{},"    \u002F\u002F Create child span\n",[165,1130,1131],{"class":167,"line":335},[165,1132,1133],{},"    ctx, childSpan := tracer.Start(ctx, \"database-query\")\n",[165,1135,1136],{"class":167,"line":357},[165,1137,1138],{},"    result := queryDatabase(ctx)\n",[165,1140,1141],{"class":167,"line":365},[165,1142,1143],{},"    childSpan.End()\n",[165,1145,1146],{"class":167,"line":378},[165,1147,1148],{},"}\n",[11,1150,1151,1152,1155],{},"Combining ",[15,1153,122],{"href":120,"rel":1154},[19]," with OpenTelemetry enables AI to automatically identify inter-service bottlenecks from trace data and generate improvement recommendations.",[29,1157,1159],{"id":1158},"backend-integration-jaeger-and-grafana-tempo","Backend Integration: Jaeger and Grafana Tempo",[34,1161,1163],{"id":1162},"jaeger-integration","Jaeger Integration",[11,1165,1166,1171,1172,176],{},[15,1167,1170],{"href":1168,"rel":1169},"https:\u002F\u002Fwww.jaegertracing.io\u002F",[19],"Jaeger"," is a CNCF Graduated distributed tracing backend. Following the ",[15,1173,1176],{"href":1174,"rel":1175},"https:\u002F\u002Fmedium.com\u002Fobservability-101\u002Fdistributed-tracing-in-kubernetes-using-opentelemetry-jaegar-a-step-by-step-guide-a48899c2b27a",[19],"step-by-step guide on Medium",[86,1178,1180],{"className":159,"code":1179,"language":161,"meta":95,"style":95},"# Deploy with Jaeger Operator\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es.server-urls: http:\u002F\u002Felasticsearch:9200\n",[93,1181,1182,1187,1196,1205,1211,1220,1228,1234,1244,1251,1261,1268],{"__ignoreMap":95},[165,1183,1184],{"class":167,"line":168},[165,1185,1186],{"class":805},"# Deploy with Jaeger Operator\n",[165,1188,1189,1191,1193],{"class":167,"line":183},[165,1190,172],{"class":171},[165,1192,176],{"class":175},[165,1194,1195],{"class":179}," jaegertracing.io\u002Fv1\n",[165,1197,1198,1200,1202],{"class":167,"line":194},[165,1199,186],{"class":171},[165,1201,176],{"class":175},[165,1203,1204],{"class":179}," Jaeger\n",[165,1206,1207,1209],{"class":167,"line":203},[165,1208,197],{"class":171},[165,1210,200],{"class":175},[165,1212,1213,1215,1217],{"class":167,"line":214},[165,1214,206],{"class":171},[165,1216,176],{"class":175},[165,1218,1219],{"class":179}," jaeger\n",[165,1221,1222,1224,1226],{"class":167,"line":225},[165,1223,217],{"class":171},[165,1225,176],{"class":175},[165,1227,222],{"class":179},[165,1229,1230,1232],{"class":167,"line":233},[165,1231,228],{"class":171},[165,1233,200],{"class":175},[165,1235,1236,1239,1241],{"class":167,"line":241},[165,1237,1238],{"class":171},"  strategy",[165,1240,176],{"class":175},[165,1242,1243],{"class":179}," production\n",[165,1245,1246,1249],{"class":167,"line":249},[165,1247,1248],{"class":171},"  storage",[165,1250,200],{"class":175},[165,1252,1253,1256,1258],{"class":167,"line":259},[165,1254,1255],{"class":171},"    type",[165,1257,176],{"class":175},[165,1259,1260],{"class":179}," elasticsearch\n",[165,1262,1263,1266],{"class":167,"line":267},[165,1264,1265],{"class":171},"    options",[165,1267,200],{"class":175},[165,1269,1270,1273,1275],{"class":167,"line":275},[165,1271,1272],{"class":171},"      es.server-urls",[165,1274,176],{"class":175},[165,1276,1277],{"class":179}," http:\u002F\u002Felasticsearch:9200\n",[34,1279,1281],{"id":1280},"grafana-tempo-integration","Grafana Tempo Integration",[11,1283,1284,1285,1290],{},"As the ",[15,1286,1289],{"href":1287,"rel":1288},"https:\u002F\u002Fwww.civo.com\u002Flearn\u002Fdistributed-tracing-kubernetes-grafana-tempo-opentelemetry",[19],"Civo practical guide"," details, Grafana Tempo is a backend optimized for storing large-scale trace data:",[86,1292,1294],{"className":159,"code":1293,"language":161,"meta":95,"style":95},"# Tempo configuration\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n    distributor:\n      receivers:\n        otlp:\n          protocols:\n            grpc:\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: minio:9000\n",[93,1295,1296,1301,1309,1317,1323,1332,1338,1347,1352,1357,1362,1367,1372,1377,1382,1387,1392,1397,1402,1407],{"__ignoreMap":95},[165,1297,1298],{"class":167,"line":168},[165,1299,1300],{"class":805},"# Tempo configuration\n",[165,1302,1303,1305,1307],{"class":167,"line":183},[165,1304,172],{"class":171},[165,1306,176],{"class":175},[165,1308,442],{"class":179},[165,1310,1311,1313,1315],{"class":167,"line":194},[165,1312,186],{"class":171},[165,1314,176],{"class":175},[165,1316,451],{"class":179},[165,1318,1319,1321],{"class":167,"line":203},[165,1320,197],{"class":171},[165,1322,200],{"class":175},[165,1324,1325,1327,1329],{"class":167,"line":214},[165,1326,206],{"class":171},[165,1328,176],{"class":175},[165,1330,1331],{"class":179}," tempo-config\n",[165,1333,1334,1336],{"class":167,"line":225},[165,1335,478],{"class":171},[165,1337,200],{"class":175},[165,1339,1340,1343,1345],{"class":167,"line":233},[165,1341,1342],{"class":171},"  tempo.yaml",[165,1344,176],{"class":175},[165,1346,491],{"class":490},[165,1348,1349],{"class":167,"line":241},[165,1350,1351],{"class":179},"    server:\n",[165,1353,1354],{"class":167,"line":249},[165,1355,1356],{"class":179},"      http_listen_port: 3200\n",[165,1358,1359],{"class":167,"line":259},[165,1360,1361],{"class":179},"    distributor:\n",[165,1363,1364],{"class":167,"line":267},[165,1365,1366],{"class":179},"      receivers:\n",[165,1368,1369],{"class":167,"line":275},[165,1370,1371],{"class":179},"        otlp:\n",[165,1373,1374],{"class":167,"line":283},[165,1375,1376],{"class":179},"          protocols:\n",[165,1378,1379],{"class":167,"line":293},[165,1380,1381],{"class":179},"            grpc:\n",[165,1383,1384],{"class":167,"line":301},[165,1385,1386],{"class":179},"    storage:\n",[165,1388,1389],{"class":167,"line":309},[165,1390,1391],{"class":179},"      trace:\n",[165,1393,1394],{"class":167,"line":324},[165,1395,1396],{"class":179},"        backend: s3\n",[165,1398,1399],{"class":167,"line":335},[165,1400,1401],{"class":179},"        s3:\n",[165,1403,1404],{"class":167,"line":357},[165,1405,1406],{"class":179},"          bucket: tempo-traces\n",[165,1408,1409],{"class":167,"line":365},[165,1410,1411],{"class":179},"          endpoint: minio:9000\n",[34,1413,1415],{"id":1414},"context-propagation","Context Propagation",[11,1417,40,1418,1423,1424,1427],{},[15,1419,1422],{"href":1420,"rel":1421},"https:\u002F\u002Fdasroot.net\u002Fposts\u002F2026\u002F03\u002Fopentelemetry-distributed-tracing-kubernetes-istio\u002F",[19],"dasroot.net"," emphasizes, the most critical aspect of distributed tracing is ",[65,1425,1426],{},"context propagation",". The W3C Trace Context standard defines the headers used:",[86,1429,1432],{"className":1430,"code":1431,"language":91},[89],"traceparent: 00-{trace-id}-{span-id}-{flags}\ntracestate: vendor-specific-data\n",[93,1433,1431],{"__ignoreMap":95},[11,1435,1436],{},"When every service propagates these headers, end-to-end traces are complete.",[29,1438,1440],{"id":1439},"sampling-strategies-and-performance-optimization","Sampling Strategies and Performance Optimization",[34,1442,1444],{"id":1443},"tail-sampling","Tail Sampling",[11,1446,1447,1448,1453],{},"Storing every trace leads to explosive storage costs. The ",[15,1449,1452],{"href":1450,"rel":1451},"https:\u002F\u002Fmarkaicode.com\u002Fopentelemetry-distributed-tracing-implementation-guide\u002F",[19],"markaicode implementation guide"," recommends tail sampling strategies:",[86,1455,1457],{"className":159,"code":1456,"language":161,"meta":95,"style":95},"processors:\n  tail_sampling:\n    decision_wait: 10s\n    policies:\n    # Keep 100% of traces with errors\n    - name: errors\n      type: status_code\n      status_code: {status_codes: [ERROR]}\n    # Keep 100% of traces over 1 second\n    - name: slow-traces\n      type: latency\n      latency: {threshold_ms: 1000}\n    # Sample 10% of remaining traces\n    - name: probabilistic\n      type: probabilistic\n      probabilistic: {sampling_percentage: 10}\n",[93,1458,1459,1466,1473,1483,1490,1495,1507,1517,1540,1545,1556,1565,1585,1590,1601,1609],{"__ignoreMap":95},[165,1460,1461,1464],{"class":167,"line":168},[165,1462,1463],{"class":171},"processors",[165,1465,200],{"class":175},[165,1467,1468,1471],{"class":167,"line":183},[165,1469,1470],{"class":171},"  tail_sampling",[165,1472,200],{"class":175},[165,1474,1475,1478,1480],{"class":167,"line":194},[165,1476,1477],{"class":171},"    decision_wait",[165,1479,176],{"class":175},[165,1481,1482],{"class":179}," 10s\n",[165,1484,1485,1488],{"class":167,"line":203},[165,1486,1487],{"class":171},"    policies",[165,1489,200],{"class":175},[165,1491,1492],{"class":167,"line":214},[165,1493,1494],{"class":805},"    # Keep 100% of traces with errors\n",[165,1496,1497,1500,1502,1504],{"class":167,"line":225},[165,1498,1499],{"class":312},"    -",[165,1501,316],{"class":171},[165,1503,176],{"class":175},[165,1505,1506],{"class":179}," errors\n",[165,1508,1509,1512,1514],{"class":167,"line":233},[165,1510,1511],{"class":171},"      type",[165,1513,176],{"class":175},[165,1515,1516],{"class":179}," status_code\n",[165,1518,1519,1522,1524,1527,1530,1532,1534,1537],{"class":167,"line":241},[165,1520,1521],{"class":171},"      status_code",[165,1523,176],{"class":175},[165,1525,1526],{"class":175}," {",[165,1528,1529],{"class":171},"status_codes",[165,1531,176],{"class":175},[165,1533,343],{"class":175},[165,1535,1536],{"class":179},"ERROR",[165,1538,1539],{"class":175},"]}\n",[165,1541,1542],{"class":167,"line":249},[165,1543,1544],{"class":805},"    # Keep 100% of traces over 1 second\n",[165,1546,1547,1549,1551,1553],{"class":167,"line":259},[165,1548,1499],{"class":312},[165,1550,316],{"class":171},[165,1552,176],{"class":175},[165,1554,1555],{"class":179}," slow-traces\n",[165,1557,1558,1560,1562],{"class":167,"line":267},[165,1559,1511],{"class":171},[165,1561,176],{"class":175},[165,1563,1564],{"class":179}," latency\n",[165,1566,1567,1570,1572,1574,1577,1579,1583],{"class":167,"line":275},[165,1568,1569],{"class":171},"      latency",[165,1571,176],{"class":175},[165,1573,1526],{"class":175},[165,1575,1576],{"class":171},"threshold_ms",[165,1578,176],{"class":175},[165,1580,1582],{"class":1581},"sOJ5S"," 1000",[165,1584,1148],{"class":175},[165,1586,1587],{"class":167,"line":283},[165,1588,1589],{"class":805},"    # Sample 10% of remaining traces\n",[165,1591,1592,1594,1596,1598],{"class":167,"line":293},[165,1593,1499],{"class":312},[165,1595,316],{"class":171},[165,1597,176],{"class":175},[165,1599,1600],{"class":179}," probabilistic\n",[165,1602,1603,1605,1607],{"class":167,"line":301},[165,1604,1511],{"class":171},[165,1606,176],{"class":175},[165,1608,1600],{"class":179},[165,1610,1611,1614,1616,1618,1621,1623,1626],{"class":167,"line":309},[165,1612,1613],{"class":171},"      probabilistic",[165,1615,176],{"class":175},[165,1617,1526],{"class":175},[165,1619,1620],{"class":171},"sampling_percentage",[165,1622,176],{"class":175},[165,1624,1625],{"class":1581}," 10",[165,1627,1148],{"class":175},[34,1629,1631],{"id":1630},"performance-optimization-tips","Performance Optimization Tips",[59,1633,1634,1640,1646,1652],{},[62,1635,1636,1639],{},[65,1637,1638],{},"Batch processing",": Set appropriate batch sizes and timeouts in the Collector",[62,1641,1642,1645],{},[65,1643,1644],{},"Memory limiter",": Configure memory limits to prevent OOM conditions",[62,1647,1648,1651],{},[65,1649,1650],{},"DaemonSet deployment",": More resource-efficient than sidecar patterns",[62,1653,1654,1657],{},[65,1655,1656],{},"Attribute optimization",": Remove unnecessary attributes to reduce payload size",[11,1659,1660,1661,1666],{},"According to ",[15,1662,1665],{"href":1663,"rel":1664},"https:\u002F\u002Fandrewodendaal.com\u002Fdistributed-tracing-opentelemetry-complete-guide\u002F",[19],"Andrew Odendaal's guide",", a well-designed sampling strategy can reduce trace data volume by 90% while retaining 100% of critical traces.",[29,1668,1670],{"id":1669},"conclusion","Conclusion",[11,1672,1673],{},"Distributed tracing with OpenTelemetry fundamentally improves observability in microservice environments. The key takeaways are:",[1675,1676,1677,1682,1688,1694,1700],"ol",{},[62,1678,1679,1681],{},[65,1680,20],{}," is a vendor-neutral framework unifying traces, metrics, and logs",[62,1683,1684,1687],{},[65,1685,1686],{},"The Collector"," enables flexible telemetry pipeline construction",[62,1689,1690,1693],{},[65,1691,1692],{},"Auto-instrumentation"," adds tracing without code changes",[62,1695,1696,1699],{},[65,1697,1698],{},"Jaeger \u002F Grafana Tempo"," integration for trace storage and visualization",[62,1701,1702,1705],{},[65,1703,1704],{},"Tail sampling"," controls costs while retaining important traces",[11,1707,1708,1711,1712,142],{},[15,1709,26],{"href":24,"rel":1710},[19]," is built on K3s with strong affinity for the CNCF ecosystem, and OpenTelemetry deployment dramatically improves visibility in microservice environments. If you are working on distributed system observability, explore ",[15,1713,26],{"href":24,"rel":1714},[19],[11,1716,1717,1718,1721,1722,142],{},"For AI-powered trace data analysis, see ",[15,1719,122],{"href":120,"rel":1720},[19]," for details. For consultations, reach out through our ",[15,1723,1726],{"href":1724,"rel":1725},"https:\u002F\u002Fwww.hexabase.com\u002Fcontact-us\u002F",[19],"contact page",[1728,1729,1730],"style",{},"html pre.shiki code .s0U2E, html code.shiki .s0U2E{--shiki-default:#F7768E}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sPY7s, html code.shiki .sPY7s{--shiki-default:#9ECE6A}html pre.shiki code .sgJMe, html code.shiki .sgJMe{--shiki-default:#9ABDF5}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sd1Qi, html code.shiki .sd1Qi{--shiki-default:#BB9AF7}html pre.shiki code .sE3pS, html code.shiki .sE3pS{--shiki-default:#C0CAF5}html pre.shiki code .sT800, html code.shiki .sT800{--shiki-default:#E0AF68}html pre.shiki code .sbD-w, html code.shiki .sbD-w{--shiki-default:#51597D;--shiki-default-font-style:italic}html pre.shiki code .sN7LL, html code.shiki .sN7LL{--shiki-default:#9D7CD8;--shiki-default-font-style:italic}html pre.shiki code .sGX4V, html code.shiki .sGX4V{--shiki-default:#A9B1D6}html pre.shiki code .sOJ5S, html code.shiki .sOJ5S{--shiki-default:#FF9E64}",{"title":95,"searchDepth":183,"depth":183,"links":1732},[1733,1738,1743,1748,1753,1757],{"id":31,"depth":183,"text":32,"children":1734},[1735,1736,1737],{"id":36,"depth":194,"text":37},{"id":49,"depth":194,"text":50},{"id":83,"depth":194,"text":84},{"id":126,"depth":183,"text":127,"children":1739},[1740,1741,1742],{"id":145,"depth":194,"text":146},{"id":155,"depth":194,"text":156},{"id":427,"depth":194,"text":428},{"id":744,"depth":183,"text":745,"children":1744},[1745,1746,1747],{"id":748,"depth":194,"text":749},{"id":939,"depth":194,"text":940},{"id":1038,"depth":194,"text":1039},{"id":1158,"depth":183,"text":1159,"children":1749},[1750,1751,1752],{"id":1162,"depth":194,"text":1163},{"id":1280,"depth":194,"text":1281},{"id":1414,"depth":194,"text":1415},{"id":1439,"depth":183,"text":1440,"children":1754},[1755,1756],{"id":1443,"depth":194,"text":1444},{"id":1630,"depth":194,"text":1631},{"id":1669,"depth":183,"text":1670},"2026-05-27","A practical guide to distributed tracing with OpenTelemetry. Covers Collector setup, auto-instrumentation, Jaeger\u002FTempo integration, and Kubernetes best practices.","md","en",{},"\u002Fblog\u002Fen\u002Fopentelemetry-distributed-tracing",{"title":5,"description":1759},"blog\u002Fen\u002Fopentelemetry-distributed-tracing",[20,1767,1768,1769,1770,1170,1771,1772],"Distributed Tracing","Kubernetes","CNCF","Observability","Tempo","Microservices","EjFERHhVPNFZpfYhIzpOnUYKy3uLQILHzlkAV5qLQVI",1779964619021]