Tuesday, 13 June 2017

From Developer to Product Owner

I have been a software developer for around ten years, but recently I was fortunate enough to be offered a different role as product owner (PO) for an Identity Platform. Keen to try something new, I accepted. However, after seven months in the role, I worked out that it wasn’t for me. This post will detail what went well, what went badly and what other developers can possibly expect if they choose to make the switch from engineering to product.


My background

I have been a developer ever since my Uni days and have worked at a number of different companies in London for around ten years. I’ve worked for Media companies, Finance and also a startup.


The Job

When I was contacted by a friend (and former colleague) about a “Technical Product Owner” role I was really intrigued. I have always felt that “pure development” roles don’t fit me perfectly as I enjoy more than just coding features. I enjoy coming up with plans, talking to a wide range of people in an organisation and taking on extra responsibility. I thought that perhaps this role could suit me perfectly but was also quite intimidated by how much of a challenge it would be. What made me accept the role was the realisation that it was such a unique opportunity that I could hugely regret declining in the future.


I took it on and was hopeful that a lot of my experience would prove relevant and that I’d pick up the other skills needed along the way.


Some of the challenges that I faced were unique to the initiatives that were in progress or planned. What I want to focus on here are the challenges that could apply to other developers making a switch to product and offer my advice.


Stakeholder management is different to working with a single PO as a developer

I was confident I’d be ok at stakeholder management. I have a lot of experience from my development days talking to non-techy POs, working out what they want and coming up with a plan of action. I assumed that as PO of a platform (with a more technical focus) engaging with the stakeholders that were non-techy and more business focused would be very similar. In some ways it is, but there are some differences that are very significant.


You need to have the confidence to dictate

As a developer, talking to your PO is all about coming to a compromise. As a PO, when it comes to dealing with stakeholders, compromise is still to be strived for but sometimes you have to dictate in the name of progress. Looking back, I wasn’t confident enough to dictate when it was necessary for a clear plan of action. I was very much aware of how important it was to be decisive, but still didn’t feel able to state “This is the plan, we’re going ahead” when I knew that they had expressed preferences that were only slightly different to my proposal.



I think this came from a subconscious feeling that I wasn’t in someway on their level. They seemed like the “real” POs to me as they could talk confidently about business terms such as revenue growth, user engagement and subscription funnels. I was more comfortable talking to the devs about techy details such as scalability, zero downtime deployments and firewall rules. Dealing with multiple stakeholders as PO in the same way that I had dealt with single POs as a developer led to too much compromise which resulted in a lot of back and forth. It took way too long for us to come up with a concrete plan of scope for our first release of a new feature.


Looking back, a key moment was when I was trying to get many stakeholders to agree on a plan of action. One day I told a more senior product owner something along the lines of “Hey, great news I just managed to get one of the stakeholders to agree with plan A”. To my disappointment he wasn’t very enthused with my update. His response was “It doesn’t matter, because we’re doing it anyway”. I thought that this was a strange thing to say at the time, but looking back I now realise its importance. In some scenarios it is impractical and inefficient to negotiate with every stakeholder because it prevents you from getting clarity on what you’re doing.


People more senior than me started to recognise the problem that I was facing. With their help a plan was put in place, communicated at a steering group and everything started to fall into place. It’s very widely known that when a team is working towards a clearly defined goal they are much more likely to get something of tangible value delivered. I have seen this first hand and realise just how important it is to have clarity from the top so that everyone understands exactly what’s needed.


Developers are used to looking at the detail - When the PO joins in, things don’t move

As a developer, I have always found estimation hard. Estimating at a high level, such as epics that have not been fully broken into stories, I find even harder. I can recall retrospectives in the past when former teams I belonged to have discussed why we think we vastly under-estimated a piece of work.


This past experience has (rightly or wrongly) made the developer in me very cautious to not overlook details that could contribute to underestimating. Other developers no doubt feel the same way.


When estimating high level epics as a team for future items on the roadmap, being super accurate is not important. Given the tendency of developers to need more detail, If the product owner supports the demands for “more analysis” or “spikes” you are left with analysis paralysis. It’s issues like this that can cause uncertainty to rumble along for too long and prevent a delivery roadmap from being formed.


A development background can lead you to present at too fine-grained detail

This sounds so obvious, but it is worth pointing out. I prepared a presentation to some non-techy stakeholders that showed user journeys and thought I was mindful enough of the business focused audience. To my astonishment, I received feedback that my message was far too detailed. Keep this in mind and get a second opinion (ideally before slaving away over your slides) to ensure you communicate at just the right level.


Stress makes you less effective

When things got stressful, being effective was a lot harder. Work felt exhausting and became miserable at times. Seeing that our “sprint planning” session was fast approaching in the calendar filled me with dread on the occasions I knew there weren’t enough stories prepared. At times it seemed impossible to create a backlog of work and switch to a more proactive, long term focus.


Seeing this issue, our delivery manager made the suggestion to put the work temporarily on hold and let the team work on whatever tech-debt they’d like for a sprint. This was a great idea as it bought us some much needed time and allowed us to finally get things in order. With stories prepared for a number of sprints, gradually our focus switched to more long term planning and things became much calmer.


A good support network is crucial

As a developer, you benefit from working closely with your peers. Pair programming, code reviews and informal chats typically occur multiple times per day and all help with knowledge sharing. This is made easier when you’re all at the same bank of desks working on the same thing.


As PO, it’s likely that you will sit with your team and far away from your other fellow PO peers. This means that the opportunities that do occur to knowledge share are highly valuable.


If a role doesn’t work out, it will still benefit you

As I mentioned at the start, the role wasn’t perfect for me. However I did gain a huge amount of experience in things that I’m hoping I can apply to other future roles. At the very least, I have a newfound respect for POs because I can honestly say from experience, it’s a tough role!

Tuesday, 30 August 2016

How to prove your Ratpack app is non-blocking.

Using the Ratpack framework can be tricky. It's easy to get something working but it's harder to ensure that your code is non-blocking.  This post will detail how to prove your code is non-blocking with automated tests.

Background - What's the problem?

In order to realise the performance benefits of the Ratpack framework you have to ensure that your code is non-blocking.  This means ensuring that your Java Threads are always busy and not waiting for I/O (i.e. non-blocking).  As soon as your thread kicks off an I/O task (e.g. sending a http request or making a DB call), it should be able to switch to another task.  Once the I/O eventually completes an event is fired which informs that the next step in the code can be performed, typically doing something with the result of the I/O task.  This means that a huge number of tasks which are I/O bound (i.e. spend a significant amount of time waiting for I/O) can be performed in parallel with only a small number of threads.  This approach can have significant CPU and memory benefits.  However, it comes at the price of increased code complexity.


In my opinion, the biggest problem is where you accidentally make your code blocking i.e. the Thread is blocked waiting for the I/O to finish and doesn't do anything useful in the meantime.  Just because you are using asynchronous libraries, there's nothing stopping you calling a blocking method by mistake.  In this situation you go back to the traditional synchronous model whereby you can only hope to achieve more parallel tasks being executed by adding more threads to your application.  What makes this situation worse with a Ratpack application is that you have invariably assumed non-bocking code and have only allocated a small number of threads to your application (the default is two threads per CPU core).  This can result in a drastic decrease in throughput.


These bugs can easily sneak through into production for several reasons.  Firstly, you can make your code inadvertently block a thread for a variety of very subtle reasons.  You don't even need to call any blocking methods yourself, to introduce blocking code.  As an example, see this code which calls Cassandra - it makes any thread which subscribes to the returned Observable block on I/O.  Secondly, when the offending blocking code is invoked from tests, unless under load, it will not only work fine but respond with the same latency as if it was non-blocking.  In this scenario, only performance tests have the potential to stop your bug going live.


How to prove your code is non-blocking with tests


In short: demonstrate that your code can handle more I/O bound transactions concurrently than it has threads. Let's assume a web application that uses standard blocking I/O, has four threads, and  perform the following:
  1. Takes a http request from a client.
  2. Makes a call to a downstream service over http which takes three seconds to respond.
  3. Once the response has been received from the downstream service, returns a response to the client.
Assuming a blocking model, the absolute best throughput that could ever be achieved is as follows:

(Time Period / (I/O Delay)) x (Number Of Threads) = Maximum throughput

Or in the case above.

((60 seconds) / (3 seconds for I/O to complete)) x (2 Threads) = 40 Transactions per minute (TPM)

In practice, the throughput would not be as high as this since other time would be required for dealing with the client's request and response.

When we consider a non-blocking I/O model, the above formula should no longer hold true since we are no longer limited by the number of threads.  

It's up to our tests to prove that the above formula does not apply to our code.  This can be done by sending N number of requests at the code simultaneously and ensuring that the throughput exceeds the maximum throughput if it were blocking.  If it doesn't exceed the throughput - fail the test!

Writing the test  


I'm a fan of writing end to end tests that prove your code can make actual requests over the network to stubbed dependencies.  Wiremock is a great tool for stubbing and mocking http services and is ideal for this test.  

We need several things in our test as detailed below:-
  1. I/O delay - To simulate a delay in our downstream http service, we can use wiremock's fixed delay feature for simulating slow responses.
  2. Constrained number of threads - We will set ratpack to have one thread only.
  3. Send simultaneous requests to our web application.
  4. Timeout - We will set the timeout to the I/O delay multiplied by the number of simultaneous requests.  If this timeout is exceeded, we haven't proved our code is non-blocking.  This can be specified quite neatly in junit tests with an annotation.
The code:
@Test(timeout = SLOW_ENDPOINT_DELAY * NUMBER_OF_CALLS)
public void handlerIsNotBlocking() throws Exception {

    URI uri = new URI(getAddress().toString() + "happy");

    List<Response> responses = new ConcurrentExecutor(
            () -> jerseyClient().target(uri).request().get(), NUMBER_OF_CALLS).executeRequestsInParallel();

    assertThat(responses).hasSize(NUMBER_OF_CALLS);
    responses.forEach(this::verifyResponseHasCorrectContent);

}
 
See github for code.


Results


When this test is run, the following happens:
  1. Eight requests are sent simultaneously to the ratpack application via a "ConcurrentExecutor" a class I created which wraps an ExecutorService. 
  2. Eight requests are sent to wiremock. 
  3. After a three second delay, all eight responses are received from wiremock. 
  4. Ratpack then returns all eight responses within around ten milliseconds of each other. 
  5. The junit test verifies that all eight responses were received ok. 
  6. The test completes quicker than the timeout, proving that the code is non-blocking. 
This same technique can be applied to other services such as Cassandra using Stubbed Cassandra. 

Wednesday, 6 April 2016

The case for keeping firewalls simple

Internal firewall rules that attempt to analyse anything higher than the network layer can cause huge problems. In this post I'll make the case for keeping your firewall rules simple.


The problem we had


Our team recently encountered an error where an internal web application received a socket timeout when trying to call one of its internally hosted dependencies.  Whilst investigating, we found that the application had made successful HTTP calls to the same service, immediately prior to the error.  



It was puzzling but I ruled out anything Network related in our investigation given:

  • The application could make some requests absolutely fine.
  • There was nothing seemingly different about the requests and responses.  They were all GET requests that returned a small amount of JSON.
  • There weren't any connection errors.

All signs pointed to the server taking too long to respond, i.e. an application issue. We then found that the system being called had no record of even receiving the failed request. It was very puzzling.  One member of the team recommended we talk to Ops about the issue.  I was convinced not to bother them as the problem must be with the application.  I was wrong!

We then found out that (through talking to Ops) that the firewall was blocking the request on the basis that the contents of the request was deemed suspect.

We wasted a lot of time investigating completely incorrect theories based on seemingly sound, but invalid assumptions.



The problem in general


How to ever know if an error is firewall related


Our problem manifested itself as a socket timeout.  How would other non http based protocols report a blocking of traffic?  I can easily imagine going through the same long learning process for a database, an FTP or an SMTP service.

Confusion is introduced - even if it never fails again


Let's assume that these problems are addressed and the logic is updated to handle the legitimate requests.  Let's also suppose that the firewall never blocks a genuine request again.  When a socket timeout error is encountered, we could now point the finger at the firewall when we should be focusing on the application.

Recommendation


I'd advocate a simple firewall for internal traffic that whitelists IPs and ports only.  If we can be certain that the firewall completely trusts traffic based on an established TCP/IP connection, things will become a lot simpler to debug.  

The simpler to debug, the quicker you fix your site in an emergency!  Time is of the essence.

If you must...


If there is an absolute requirement that these firewall rules are in place, confusion can be mitigated by performing the following:

  • Ensure that all developers are aware of how firewall issues may present themselves.
  • Provide a console for everyone to easily see if traffic is being blocked by the firewall or not.

Tuesday, 8 March 2016

Ratpack talk - The story so far

I recently gave a talk to developers at Sky where I am currently working with Energized Work on the subject of our team's experience with Ratpack.



Apologies for the technical issue half way through (and also for the huge number of erms and ers). 

Slides: http://www.slideshare.net/PhillBarber/ratpack-the-story-so-far




Thanks to everyone who came along at Osterley.





...thanks also to the people watching from the Leeds office.

Tuesday, 23 February 2016

Lesson learned with Ratpack and RxJava

We recently encountered an issue with integrating Ratpack with RxJava's Observables.  This post will detail what the error was, how it was broken and how it was fixed.

The code 

Essentially, our handler code was as follows (NOTE: see git project for this class and also a test showing the behaviour of the broken code):

@Override
public void handle(Context context) throws Exception {

    Observable<String> contentFromDownstreamSystem = observableOnDifferentThreadService.getContent();

    contentFromDownstreamSystem.subscribe(response -> {
        context.render("Downstream system returned: " + response);
            }
    );
}


The code above retrieves an Observable from a service.  As implied by the variable name, the Observable emits items on a different thread when it is subscribed to.  The Obsevable I have used to illustrate this issue is very simple and can be found here.  In our real world example, our Observable represented a ResultSetFuture from Cassandra which would emit items after a (very short) period of time.  

The errors


No response error


The first error we encountered was as follows:

[2016-02-17 08:39:16,812] ratpack-demo WARN  [ratpack-compute-1-2] r.s.i.NettyHandlerAdapter - No response sent for GET request to /observable-different-thread-broken (last handler: com.github.phillbarber.scenario.observablethread.ObservableOnDifferentThreadHandlerBroken)


Despite the fact our Observable was emitting items that would take some time to be emitted, the above error occurred seemingly immediately after our handler had completed.  This also resulted in a http 500 error response issued to the client.

Double transmission error


It gets better!  Not only did we get an error indicating no response, we then saw an error implying we had tried to send two responses as follows:

[2016-02-17 08:39:16,821] ratpack-demo WARN  [Thread-4] r.s.i.DefaultResponseTransmitter - attempt at double transmission for: /observable-different-thread-broken
ratpack.handling.internal.DoubleTransmissionException: attempt at double transmission for: /observable-different-thread-broken

The Problem


The problem here is that Ratpack is not aware that the request is dependent on the Observable emitting items.  In other words, the request's Execution does not contain a reference to the Execution segment which represents the Observable's success action (the lambda passed to the subscribe method).  Since it seems to ratpack that there s no further work to do, ratpack's NettyHandlerAdapter detects that no response has been sent and issues the "No response sent for request" error and issues an error response to the client.

The final twist is that eventually our Observable's action is completed.  When it tries to write a response, it can't as the response for the request has already been committed.  This is why we get the "double transmission" error.


The fix - Convert your Observable to a Ratpack Promise


We need to ensure that the request's Execution has a reference to the Execution segment of our success action.   This is done by converting the Observable to a Ratpack Promise and activating it as follows (see fixed code in git here and a test here):

@Override
public void handle(Context context) throws Exception {

    Observable<String> contentFromDownstreamSystem = observableOnDifferentThreadService.getContent();

    RxRatpack.promise(contentFromDownstreamSystem).then(response -> {
                context.render("Downstream system returned: " + response);
            }
    );
}


If you read the ratpack documentation, this will seems obvious and you might wonder why we made this mistake in the first place.  However we were tricked into thinking that the broken code would work due to some very subtle ways in which the broken code can actually work.  The broken code will work just fine under the following scenarios:

  1. If the Observable returned by the service was converted from a Promise e.g. a Promise returned by the Ratpack httpclient.  That way the Promise will have been activated (or rather the execution segment added to the execution) indirectly by some other code and not explicitly by the Handler.
  2. The Observable synchronously emits items on the same thread.  Not sure of a real life example as to why you'd do this but it can occur during your testing when mocking and stubbing.
Our conclusion is, that when using RxJava with ratpack you should always convert to a promise in your handler layer.  You should do this even if you don't need to (as descibred by points one and two above) so as to play it safe incase the implementation of the Observable changes in the future.  

Why are we using RxJava


When we decided to use Ratpack, we wanted to avoid depending on it throughout our entire codebase.  If all of our services dealt returned Ratpack Promises, we'd have an even bigger job on our hands if we decided to switch frameworks.  It was hoped that using RxJava would decouple most of our code from Ratpack.  

Even with the extra learning curve of using RxJava, this seems reasonable as typically a web framework will only be referenced from your code in the front end, web/controller layer. It seems a bit of an anti pattern to do depend on it throughout the entire code base.

Summary


You only understand how things work when things go wrong.  This was a great problem for us as a team to figure out since it taught us about the intricacies of how Ratpack actually works.

  

Sunday, 10 January 2016

Choosing between Ratpack and Dropwizard.


This post will discuss our team’s approach to evaluating the java based web framework ratpack as an alternative to dropwizard for creating a new microservice.  This post assumes a basic prior knowledge of both ratpack and dropwizard.

Background


The team I'm on is full of developers all with experience of creating and maintaining applications built with the Dropwizard framework. It’s safe to say that Dropwizard would be the team’s default preference when considering frameworks for web based java apps. However, we have recently been tasked with building a new microservice that is as efficient as possible. Although this non-functional requirement isn’t 100% specified, we do know it will need to be capable of high throughput and low resources (i.e. memory and CPU). High costs from the company’s PAS provider have in no doubt influenced this requirement.

Ratpack is said to allow you to create "high performance" services that are capable of meeting these non-functional requirements assuming that your application is I/O bound.  It's because of this that the team evaluated it.

Dropwizard vs Ratpack performance


A Dropwizard vs Ratpack performance test was conducted to see if the reputed performance benefits could be observed for our use case.

Two webapps that performed the same steps (detailed below) were created, one with dropwizard the other with ratpack.

Architecture Diagram




Load Profiles  


Three load profiles were given to each application as detailed below:
  1. 5 minute duration, 100 JMeter Threads
  2. 5 minute duration, 200 JMeter Threads
  3. 5 minute duration, 300 JMeter Threads

Results


The third test run resulted in the Dropwizard application eventually failing to respond.  


The graph above shows that throughput for the first two test runs was very close.  The third test run shows Ratpack answering significantly more requests per second due to the fact that the Dropwizard application began failing to respond.  Exactly why the Dropwizard application gave up was not investigated. 





Here we see ratpack using significantly less memory in all but the last test run. 






Ratpack seemed to use around half of the CPU as Dropwizard.




Here we see the true nature of our non-blocking frameworks in action.  Ratpack used just twenty threads in each test run.  Knowing the basic premise of non-blocking I/O, this shouldn't come as a surprise - however I still find this very impressive!



Performance Summary

Ratpack can clearly handle more requests with less resources in an application that spends most of it's time waiting for I/O.  This was of major significance to us since we knew from the start our application would be I/O bound.  It's also worth pointing out that (at least in my experience) I/O bound webapps are very common.

Other concerns


In order for the organisation to support our new service, it needs to comply with a few standards.  

Logging

A common design design pattern in a microservices architecture is to assign some kind of UUID to each request which is then passed to downstream systems.  This  UUID can then be added to log events so that they can be correlated across multiple systems.  This can (and has at our organisation) been implemented using SLF4J's Mapped Diagnostic Context (MDC) which "...manages data on a per thread basis."  However, it should be noted that " a server that recycles threads might lead to false information".
Ratpack recycles threads, but luckily were covered an MDCInterceptor has been created which addresses this very problem.

Deployment


The organisation relies on the simplicity that comes with building and deploying Java based services as Uber/Shaded/Runnable jars.  Since ratpack applications can be built in just the same way, this wasn't an issue.

Integration with Hystrix


Hystrix is a great library that helps you make your services fault tolerant.  Not only is it great, it's become the unofficial standard within our organisation.  Any java based application can integrate with Hystrix in many different ways since it's so flexible.  Hystrix supports synchronous, asynchronous and reactive programming models.  Not only does Hystrix support non-blocking calls it also supports the use of semaphores instead of Thread pools to manage concurrent downstream calls.  The flexibility and support offered here by Hystrix is absolutely crucial!  If Hystrix mandated the use of a thread pool or blocking calls, we'd be back to the Dropwizard performance characteristics (shown above) of having one thread per concurrent request we handle (assuming each request results in a downstream system call, which is true in our case) which would negate ratpack's benefits almost entirely.

Not only is it possible to use Hystrix with ratpack, it's also easy to use features such as request caching, request collapsing and streaming metrics to the dashboard (which is awesome by the way) with the ratpack-hystrix JAR.

Complexity (Learning Curve)


The benefits of ratpack come at the cost of complexity.  Our team (myself very much included) are used to building traditional synchronous based Java apps.  The effort of adopting an asynchronous programming model was difficult for the team to assess but was certainly a known risk which could affect delivery time.

Decision Time - Ratpack or Dropwizard


With the amazing performance from our benchmark, a sense of optimism that we’d pick up the new framework and asynchronous programming quickly... the decision to go to ratpack was made.



My next post will detail the team's experience in general with Ratpack. 

Thanks to the team for all of the shared learning so far, and especially to @mirkonasato for leading the performance benchmark work detailed above.

Saturday, 31 October 2015

Ansible's for loops

Recently, me and a team hit a big limitation with Ansible that really took me by surprise.  For loops, can only iterate over a single task and not roles or playbooks.  This post will discuss why it was a surprise, what we did to get around it and (briefly) what Ansible 2 introduces.

Pre Ansible 2


The original issue really took me by surprise since the documentation on for loops showed a huge number of possibilities (including conditionals, looping over arrays etc etc) that suggested to me it was Turing-Complete and easily capable of meeting all our looping needs.  So when I needed to assign a role multiple times (dynamically from a list) I was shocked to find that it couldn't be done.

I'm not the only one who was caught out by this.  See here, here and here.

Why this wasn't supported really puzzled me.  Some on my team concluded that it must be due to the way Ansible works internally and would perhaps require a lot of re-writing to support the feature.  However, Michael DeHaan said here that there was a "fundamental reason" it couldn't be done:

"...the role must be applied in the same definition (but not the same variables) to each host in the host group. Ergo, you can't have different numbers of roles applied to different hosts in the same group based on some kind of inventory variance."

But I don't understand why you can't.  The hack here suggests to me that you can.

Should the docs have told me what's not possible?


I also wondered if the ansible documentaion on loops should have mentioned its limitations.  Funnily enough, Michael DeHaan wrote an article on Technical Documentation As Marketing which said "your documentation and front material is there to both be honest, communicate how to use something, and also sell what you are doing".  The docs certainly didn't lie to me, but they could have been more honest about limitations.

Workaround

We looked at a few options:

  1. Hack - Way too ugly!
  2. Loop over each task within a role separately - This was not possible for us since some tasks depended on output from previous tasks. 
  3. Convert role (or set of tasks) into a single task - No appetite to learn and code more ansible specific stuff at a time when we felt so let down by it!
  4. Move logic away from ansible into something external that ansible calls (e.g. a bash script).  
We opted for the least bad, which to us was option four.  


Post Ansible 2

According to slide 15 here, in Ansible 2 you can loop over multiple tasks which is really good to hear.  However, it seems you can only loop over tasks, and not roles.  I don't understand why you can't loop over roles, but am really encouraged to see the feature included.  It's also great to hear that Ansible 2 is backwards compatible with version 1.  I look forward to using it.