Using Jaeger Cpp Client for Distributed Tracing in Ceph

If you are not already familiar with Jaeger and Opentracing, you can read about it here.

Jaeger + Opentracing provides ready to use tracing services for distributed systems and is becoming widely used, de facto standard because of their simplicity and standardization.

Jaeger, the backend provided to the Opentracing API is responsible for the collection of spans, these spans are made with use of smart pointers that carry the timestamp, TraceID and other meta info like a specific tag/log associated with the span to uniquely identify it across the distributed system.

let’s get intuitive using Jaeger-client-cpp

I would suggest going through the excellent walkthrough of the HOTROD application before we begin …

Okay, you are back! Let’s get started!

To make use of Jaeger, follow the README.md and test the spans build in the example app. You’ll get to see two spans, by following the example App.cpp.

INFO: Initializing logging reporter
INFO: Reporting span 5b163b107a813958:a0e81b958c00c461:5b163b107a813958:1
INFO: Reporting span 5b163b107a813958:5b163b107a813958:0:1

The first INFO line specifies, that we have the reporting service enabled and hence the spans would be reported locally when you run your traced application. The second and third INFO are actually the spans generated.

These spans can be seen in the Jaeger UI(you have to first initialize UI before running the traced code to see spans).

Use their all-in-one docker image(you will need docker installed for that, head here if you don’t already have docker) for hassle-free UI setup.

How traced spans looks on Jaeger UI.

Show me what Opentracing API along with Jaeger is capable of!

For that, you can refer tutorial-example-cpp that illustrates the use of Opentracing API. I can give a brief overview on it.

Configuring Jaeger
Jaeger comes with an option of setting up the tracing system according to the requirement. By this, I mean you can customize capturing spans based on your need.

You can set if you want to enable logging, set the frequency of recorded spans by using samplers provided by Jaeger, do you want to persist those spans like for maybe billing you want to record the time spent on a service using tracing, for instance, if it’s a microservice like Netflix and you want to collect and store all the timestamp that spans carry so as to bill your customers you can forward your spans to database storage units.

Configuration options for jaeger -client-cpp can be passed in the form of YAML file. An example of which can be seen here.

Setup the Tracer

To employ tracing to your code first step is to choose a Tracer backend compatible with Opentracing API, you can find available ones here.

We are going to use Jaeger and hence will replace mocktracer, with Jaeger specific initialization, and pass the config.yml file location for our customized configuration, after which our job with Jaeger is done!

From there Opentracing API is used to configure and add traces which are then collected by Jaeger backend and sent to Jaeger UI. Replace the mocktracer with Jaeger and we are good to go. You can make use of this code.

void setUpTracer(const char* serviceToTrace) {  
auto configYAML = YAML::LoadFile("config file location"); 
auto config = jaegertracing::Config::parse(configYAML); 
tracer = jaegertracing::Tracer::make(serviceToTrace, config, jaegertracing::logging::consoleLogger());
opentracing::Tracer::InitGlobal(      std::static_pointer_cast<opentracing::Tracer>(tracer));
} 

Parent, Child and Follow-up spans:

After we had setup the Tracer we should start creating spans to trace our application, for that taking inspiration from Jaeger-client-cpp example I have added a function that converts the context passed as a string to a span, this method returns a span which can be stored and passed to create subroutine context.

The parent span so obtained from tracedFunction() method is then used to pass into tracedSubroutine() function along with subroutine context, which creates child span.

Note: there can be only one active parent span in scope, either the span finish when the scope ends or you can use SpanName->Finish () method to destroy it explicitly.

Tags and Logs:

The purpose of tracing is to give as much meaningful contextual information in an easy to comprehend form; as Google developers describe it as making things observable.

We can add more context to our tracepoint using Tags and Logs, refer docsknowing more about them.

Span Context propagation by Inject and Extract methods:

Since we are working on distributed systems, with many remote machines coordinating with each other concurrently exchanging data, messages, updates; it becomes important to trace these subparts of the system as much as the core system itself.

Thus, Opentracing and Jaeger also supports off the wire context transfer, by which we mean the span context is gathered using the Inject method, the string so returned will be carrying the span context and hence the uniqueness of a span, which can then be transmitted and reproduced back to a span ready for tracking at the remote system, if Tracer is already not active initiate it at remote end and the use extract method to reconstruct the span.

Voila! And the span’s reconstructed back!

Marking those first traces and making it more intuitive to add traces.

After a few hiccups with the use of unique pointers and adding jaeger libraries locally. I can now build Ceph with Jaeger!

I tested adding the initial span; in OSD input-output path(a part of Ceph) and after a bit tussle understanding how to make it work, I can now see the traced path by adding just 1 span! No wonder it would be great to design useful tracepoints that can hopefully reform the way debugging is done in distributed systems like Ceph.

let me show you how those spans turn out for Ceph Object Storage Deamon(the one who manages and stores objects in Ceph cluster)

Final Words

The traces marked acts as a proof of concept that Jaeger can be employed efficiently in large distributed projects like Ceph.

My future challenge lies in providing meaning to this powerful tool, by designing the best outline for adding tracepoints, which requires a thorough understanding of Ceph, which is where I am beginning from. Yet there will always lie the need for tracing for new code or some suspicious code at unfamiliar places, keeping all that in mind, I have abstracted Opentracing API in easy to use functions that can by anyone so that developers have to facility to add new traces themselves easily.

Making this system more decentralized along with a strong foundation of meaningful and general tracepoints, is where I want to bring this project to.

Let’s see what lies ahead!

Useful References:

Advertisements

Jaegertracing in Ceph 101 and my struggles till now.

Hey there! o/

I am interning with Ceph for Outreachy Summer, 2019. You can read about my experience applying here.

To begin with I must say this is a realisation, there are just two states to developers.

Image result for two states of developer meme
Credits: google images

Working with large codebase like ceph. My first conversation with my mentor was how should I go by studying this overwhelming codebase.

He then gave me some outline on where to begin. How to approach, moreover I got to learn the most, discussing and seeing how they see things, one of which was catching an error I was having while setting up my environment.

According to my mentor Josh Durgin,

An engineer must know how to read a codebase, it is so important that you can reach the call stack and debug at much deeper level.

As we enter the third week, I am gonna share how it has all been along with my experience of how to read Ceph source code and troubleshooting issues.

I’ll describe it further, after a brief outline about the project, so that you get the context.

What am I working on?

TL;DR I’ll be working with Ceph on adding jaegertracing tracepoints, which will make Ceph more robust and easier to trace and hence debug

What is Ceph?

Ceph is a large scale distributed storage system, that intelligently handles Peta byte scale of data, it is based on RADOS ( Reliable, Autonomic Distributed Object Store ). Ceph is a huge codebase, with passionate people working on fast paced development, with daily report meeting/ discussion, community events, hackathon and an annual meet Cephalacon. Read more about Ceph here.

What is distributed tracing?  

With the increasing complexity of the existing system the need for debugging these complex systems arises, these can’t be sufficed with the mere print statements and tests. They need to have discrete distributed system of tracing. A very good example of which could be seen in this picture, tracing a monolithic service makes things invisible even if one system fails all further operations can’t be traced, which fails the whole motto of tracing. So the need of easy to integrate, discrete, standard and capable of generating insights tracing system arose in cloud community, but with no standardization, no system could provide the wholesome functionality that were needed.

You can learn more about tracing reading about Dapper, Lightstep and Zipkin

Why Ceph chose to use distributed tracing?

With the coming of standard libraries that are well maintained with active contributors. like Jaeger, the distributed tracing technology that was existing since a decade, but could not be used because of ease of standardization can now be easily integrated. Ceph can reach to a much improved tracing system, as Ceph has a lot of background processes which the existing Lttng, Blkin, Babel trace and Zipkin compromised system does not do justice to and has a learning curve without much insights, contrasting to which Jaeger presents a nice UI representation of traces, which can then be used to gather insights.

If this could successfully be achieved, it could transform the way Ceph is being debugged. Hence, making “Ceph faster in face of failure”

This was the basic overview of the project now talking about the problems I faced till now:

Setting up the development environment

The most troublesome step in the whole process was building the project, although I had earlier build Ceph before applying, but it didn’t include the Jaeger Libraries that Ceph would need to add instrumentation, first thing was to figure that out. I cleaned that code and locally installed the libraries, yet there were some issues that came up; I tried fixing them in dependencies code, but have to resort to commenting them from source and rebuilding as they were just some check, that did the trick.

Vstart troubleshoot

The most significant problem I came across was due to some conflicting code in Ceph, my build even after being successful was not starting, then my mentor oriented me about how to track that error down. I not only solved the issue but also learned the whole process of debugging Ceph, which I’ll add extensively in coming posts hopefully.

Communication and working in different time zones

My mentors belongs to EST time zone and I to IST, initially I had problem adjusting to EST time zone but I now prefer working those hours as; whenever I have an issue I would not have to wait a day to discuss it, I leave a message on slack and if I am still not able to solve it, then I ping my mentor, this way they also can be aware of what I am doing.

Asking for help and why Ceph is awesome!

Working remotely teaches you the value of communication, it’s when you get stuck and want to just bang your head, you reach out. The great thing about Ceph is it’s community, they are a bunch of compassionate people who are there for you, with communication being backbone for their growth. I participate in daily meetings where we update about our progress and discuss issues if any, along with this I have really supportive mentors, Josh, Neha, Ali with whom I get my doubts over audio/video, irc, chat and mail.

My experience had been awesome till now and I hope it continues to be same, I’ll write about where to look at when troubleshooting Ceph and write more about my experiences in coming posts.

How to apply for Outreachy, being a complete beginner! A Step by Step Guide.

Hey there! o/

So you found this post and I suppose you are looking for some guidance on how to apply for Outreachy.

Cheers to you for deciding to make your Summer/Winter break awesome! \o/

A little bit about me, I am Deepika Upadhyay and I’ll be interning with Ceph( an open source massively scalable storage solutions) on the project “Instrumenting the OSD with end to end tracing” this summer.

I had broken down the whole process in steps. Walk with me, we have got this and we’ll get you through application in a breeze. 😉

I’ll consider you a complete beginner; if you are, which is perfectly fine and if you are not, you’ll still learn a lot! 😉

Let’s begin!

What is Outreachy and why you should apply for it?

credits: Planet Outreachy
                                              

So what is Outreachy by the way?

Outreachy is a paid internship program that provides interns to work on Free and Open source Software(FOSS), remotely from any part of the world.

If you identify yourself as a minority(Anyone who faces under-representation, systemic bias, or discrimination in the technology industry ) and want to work with FOSS while getting paid, mentored and connecting with awesome people, then Outreachy is for you.

The interns are paid a stipend of $5,500 USD for the three-month internship and have a $500 USD travel stipend to attend conferences or events.

Being sometimes misunderstood because of the similar program Google Summer of Code(GSoC), it is not limited to students and neither just to programming; it has a variety of projects ranging from programming, user experience, documentation, illustration, graphical design, data science and more. And unlike GSoC, you need to make at least 1 relevant commit and 40hrs/week for 3 months availability to be eligible.

What motivated me to participate?

My interest in open source started with being part of the Durgapur LUG(summer training experience) and my college’s developer community. During training, I was searching for opportunities in open source when I came across Tapaswini Pathak’s Open Source program’s list.

I explored more about Outreachy by reading experiences shared in Outreachy’s website of their interns’. It instantly struck a chord with me and at that moment I knew, I wanted to become part of this program.

My Experience of applying twice!

I was a junior when I first applied for Outreachy, a complete beginner. Determined yet inexperienced … I did a fatal mistake, as I was interested in many projects and when you are choosing so many projects you are deciding upon dividing your attention to all of those initially demanding tasks as a beginner, I’d say you should narrow down your choices to at most 3 projects. Ideally, select 1–2 similar tech stack projects, the ones you are really really interested in and can learn/have basic skills as you to go with.

Applying this time, I understood the process better, I went with selecting the one project I was really interested in; communicated with my mentor and tried to understand the project more and more, along with working on making the initial commit and the application.

I tried making things easier for the evaluation process by well documenting the things I worked on, which also served me greatly when I got back to pick things up, after exams.

Yeah, I did feel confused sometimes, but I was motivated to become the part of the community this time for a long run, learn from my mistakes; apply with a strong application and just get started!

How to select a project?

As earlier said, it’s the most crucial step in the whole application to select the project you feel right fit for you, after careful analysis.

The outreachy website has a description of projects, along with tech skills requirement and mentors contact, read them and ask yourself:

Does this project interest me?

Would I be able to adapt/be the right fit for tech skills that it demands?

What purpose does doing this project serve for me?

Maybe also this, Am I willing to be a long term contributor for this project and give back to the community?

If you are confused or couldn’t narrow them down as a beginner; reach out to mentors or ask for help from some experienced, the best thing about open source community is they are incredibly welcoming bunch of people and I’m sure they are gonna help. You can always reach out to me, I’d love to help! 🙂

Learnings from the application process

Thanks to the organizers for well documenting and organizing the steps clearly in the application process.

Be an early bird and start working right after the announcement of projects, but even if you had discovered a little late, things could be worked out, although chances are the slots for projects that interests you were already taken.

  • Once you have chosen a project, you should get in touch with a mentor.
  • Learn the basics of git or whatever the initial application task demands.
  • Setup the development environment( do not underestimate this step, it could take considerable time sometimes)
  • Find a task relevant to project to work, stay in touch with your mentor
  • Discuss with your mentor and prepare your application, get it reviewed, apply!

At last, I see the whole process as more like a plateau than a linear curve, so if you aren’t able to see any results immediately, just hang in there!

Google, ask your mentor, discuss on forums, but do not give up!

After all the phase of selecting the project, working on the initial commit, reading, learning, reaching out and failing time and time again, once you get by this phase, the next commit and the further process becomes easier.

Consistency and communication matters a lot, a commit, a day for 10 days, is always better than 10 commits in a day.

Final Words

I would not say that it was all cakewalk, there were times of frustrations, getting stuck, nights going by without you even being aware of time, and so on, but when you see the learnings from it, all of it seems totally worth it.

It’s been a wonderful journey till now, participating in Outreachy, I had felt intimidated being a complete novice sometimes, somewhat sound the other times; perplexed, stagnated, imposter, doubted myself time and time again, but what kept me going was that I love open source and the thought, that every expert was once a beginner, what matter s is its my journey and I want to be the best version I could be.

I would like to thank, Ali Maredia my mentor from Ceph for always being there helping out in application process; my college developer’s community and DGPlug, I learned a lot from you people, Outreachy Organisers for the concern they show for each applicant whether they are selected or not, guiding us from the very start and helping us by providing opportunities during and after internship, speaks for the quality program it is!

Advice for outreachy organisers

Since the organisers have reach to awesome people and conferences in the industry, it would be nice to see the interns for sharing their experiences and speaking up(some exceptional interns hesitate to do that initially or can’t find opportunities being beginners) that would even serve the purpose of spreading word about the kind of people, work and projects there are in Outreachy. It’s a wonderful program and I’d love to see its roots deepening more and more in welcoming diversity.

Using BaaS, GraphQL and React for making high performance apps

  • What are we targeting?

So you wanted to make a highly efficient app, with fast processing of requests. In this post, I tried to narrow down the app designing options, that could make your app loaded with all the superpowers.

We are going to build a Demo Application since we want to develop a hassle-free, easy to maintain the app, we narrowed to following choices to select the app designing with.

 

  • Outline for designing our application model API.

 

  • Backend as a service(BaaS) (Parse-graphQL-platform/ServerLess): As using BaaS makes it easier to maintain server, as it reduces the expertise required for maintaining server-side code, now we just need to worry about deploying our service onto the server without worrying about server maintenance, from all the available BaaS such as Microsoft Azure, AWS Mobile App Backend, Firebase, etc. we chose to use Parse Server, as it has an advantage over others with its ease of use, dashboard for monitoring, ability to control databases, application layer, etc.

A good example for this will be TodoMVC application, we want to run our Parse (web/mobile SDK) coupled with Parse Server+Parse Dashboard running on Express using GraphQL and REACT Native based mobile client, the Parse models would have resolvers using Parse ACL system for authentication, making our app more secure.

#Need to add further to this post

 

Why I fell in love with the open source world!!

This is just what I feel, so just take it lightly.
Being a part of Open Source feels like I have entered a whole new world, I had teleported to the developing world where continuously something is getting created.
When I used to think of some practical implementation of a thing that mere thought that it could be done was not so easy to even imagine with the amount of work and research involved, but when you enter a world where this is the primary motto of doing things and you see people driven to help, putting their hand forward to support you … It feels like dreams do come to reality!
With the magical world not yet discovered but present around you,  I would suggest my future self, do diversify your connections, you might never know another new world might be waiting for you … I came to know about open source from one of my college senior last year because I participated in a debate, and when you form a neural connection of these worlds, something amazing, something out of the box will get evolved!
After all, it’s all about connecting the dots! 😉
It seems like I can overcome that state of monotonous cursing to have not accomplished learning and that turned it into effortless task cause when you build things learning comes as a by-product.
Learn from people especially their failures cause they will elaborate and give reasons for why they failed, but on success maybe this might be not so easy for even them to find a thing that worked cause it’s a mingling of many
Hear out some interesting people you will make more discoveries of how they see the world ask a poet, what is there in the words that are beautiful, ask an algorithm how he sees algorithm in daily life , ask a painter how he creates a beautiful masterpiece with just a simple pen and the back of his notebook, ask how a musician visualise the tones, which kind of music gives him goosebumps.
 That’s how you can taste life in different forms, reading books, hearing tales, creating a workaround out of a failure cause success doesn’t taste as good as rising from hardship does.  🙂

Importance Of Critical Feedback

Somedays I wasn’t able to cope with college-hours and extra-curricular and wanted to start to cover-up everything that I missed these days, looking back from where I started, 3 months prior to DGPlug summer training, I feel this was the most helpful in growing my understanding for free and open-source world.

FEEDBACK SESSION- BY SANKARSHAN

I think it’s very important to have a feedback loop, where you’re constantly thinking about what you’ve done and how you could be doing it better. I think that’s the single best piece of advice: constantly think about how you could be doing things better and questioning yourself.

Elon Musk

Earlier,  I had an understanding that feedback is quite essential in life , I write dairy to maintain this feedback loop to improve self  but what I learned from this discussion was that , we need to give and take critical feedback in a balanced form what this means that if you be are  presenting much of critical details to someone, maybe the receiver might not take it well contrasting with that, being too abstract like the “the product looks awesome ” would not contribute to any improvement of any sort .

Another point I found interesting was that while giving or receiving feedback we must not be too much emotionally invested, as this might make us somewhat bias, which again fails the purpose of feedback which, in my perspective is,

The step that is mostly ignored in the development of anything be it business, code evaluation or skill development!

And yes, I do believe feedback can accelerate the development several-folds if received from a person having experience in the art or by self -evaluation by time travelling and going in past to see where you went wrong, or reading and gaining knowledge in an unbiased manner, just like how a case is evaluated in courtroom for coming to a right judgement, we need an unbiased evaluation similar to that and being tolerant to take one too.