JFDIAnna Shipman's blog2024-03-09T10:07:54+00:00https://www.annashipman.co.uk/Anna ShipmanLearning from a strategy project2024-01-14T00:00:00+00:00https://www.annashipman.co.uk/jfdi/learnings-from-strategy-project.html<p>A couple of years ago I worked on a strategic project that was outside my remit. It was an incredible learning experience for me and here is some of what I learned.</p>
<h2 id="the-problem-i-wanted-to-solve">The problem I wanted to solve</h2>
<p>The problem for my team was that our tech estate was in contention. I was leading one of a number of engineering groups within a larger organisation; each group had its own priorities, but most of them required delivery through my team or tech estate; and we had our own priorities. So we ended up slowing each other down.</p>
<p>It seemed to me that we needed an overarching strategy to enable us to resolve these conflicts and all pull together towards the same ultimate goal, but it didn’t feel like my place to work out what that would be: it would impact my peers’ teams as well as my own so it felt like the solution had to come from the level above.</p>
<p>However, two things changed for me. I started working with an excellent coach, who helped me unblock my thinking. Then, I was talking to someone about how it wasn’t for me to solve this problem and they said: why not you?</p>
<p>And I thought, yes, why <em>not</em> me? So I pitched to my boss that I step aside from my role for three months to propose a solution.</p>
<p>This post is not about the answers I came up with, but about what I learned through the process. And it was a lot.</p>
<h2 id="what-i-planned-to-achieve">What I planned to achieve</h2>
<p>I had planned to work out our strategic direction, including any changes we’d need to make to our technical architecture or organisation.</p>
<p>My proposed end goal of the three months included:</p>
<ul>
<li>a plan for what technical work would be required and roughly how complex it was</li>
<li>how might the desired end state fit with current funding models</li>
<li>what the cost impact would be</li>
<li>a team for initial changes ready to start work</li>
</ul>
<p>I had also hoped that we would reach these goals by clarifying the overarching strategy.</p>
<p>This sounds laughably ambitious now, but when I proposed this piece of work, I genuinely thought I would be able to get all or most of that done in the agreed time.</p>
<h2 id="what-i-actually-achieved">What I actually achieved</h2>
<p>I adddressed the main issue in a little longer than intended: after five months we had a clear technical direction for how to address some of the contention on the codebase and a team starting work on it.</p>
<p>However, I hadn’t addressed funding models, I hadn’t brought everyone along on the journey, and I definitely had not resolved the underlying contention between groups about the strategy.</p>
<h2 id="what-i-learned">What I learned</h2>
<p>This was an incredible learning experience. Here are some of the things I learned.</p>
<h3 id="dont-wait-to-be-asked-you-can-put-yourself-forward">Don’t wait to be asked: you can put yourself forward</h3>
<p>At every level of my career I’ve learned that you don’t have to wait for someone else to solve a problem you have, or to create the opportunity you believe exists. However, each time it looks a bit different. This time it looked like I would be treading on toes and/or would not be in a position to be able to achieve results.</p>
<p>If I had just gone off and done it on my own, it would have been difficult. However, this work solved a problem for my boss. Pitching how I could get us to a solution meant that he backed me to do it, which conferred the authority to get the work done.</p>
<h3 id="put-the-work-in-to-clarify-expectations">Put the work in to clarify expectations</h3>
<p>In my enthusiasm for the opportunity, I didn’t clarify exactly what I hoped to achieve.</p>
<p>This led to two main problems:</p>
<ol>
<li>People started to think this piece of work was going to solve all things for all people</li>
<li>The exec team didn’t understand where we were in terms of progress</li>
</ol>
<p>I didn’t clarify, because I didn’t know all the possible outcomes. But this is a problem that has been solved in engineering. In this situation, either be really precise about the outcome you plan to achieve and then work towards that, updating on progress as you go, or timebox the efforts and report on how far through that timebox you are.</p>
<p>It’s quite easy to sell a poorly defined piece of work that will solve everyone’s problems, and is much harder to sell a piece of work that is more clearly defined but smaller in scope. However, it’s worth putting that work in at the beginning to make sure all stakeholders understand what the goal is.</p>
<h3 id="its-very-important-to-know-who-your-stakeholders-are">It’s very important to know who your stakeholders are</h3>
<p>This is good advice in general, but with a piece of work like this it’s crucial, as your results dependend on buy-in. In this case, a peer of my boss was a key stakeholder. She already had a solution in mind, and saw my work as part of that, which led to crossed wires when I took a different direction than the one she had anticipated.</p>
<p>The learning for me was that it’s not always obvious who the key stakeholders are and it’s important to understand that and manage them closely.</p>
<h3 id="if-you-want-to-go-far-go-together">If you want to go far, go together</h3>
<p>This was not just an engineering problem, so I asked two of my colleagues in product to work with me on this. This was great, as their understanding and challenge helped the work be much better.</p>
<p>We also talked to a wide range of people to get their input. We conducted about 50 internal interviews and I also had about 10 conversations with external companies to understand how they did it. Working as a team with other people made it possible to have that many conversations, and bought a lot of learning and diverse thinking.</p>
<h3 id="you-need-to-bring-everyone-else-on-the-journey-with-you">You need to bring everyone else on the journey with you</h3>
<p>The most important thing I learned from this piece of work is: it’s not enough to find the right answer. In fact, finding the right answer is really not that valuable. What is valuable is creating change; and in order to create change, you need to bring people along with you.</p>
<p>We did bring people along on the journey with defining the problem. When we’d completed internal interviews we did a very well attended talk outlining the challenges we wanted to address and people were clearly engaged. However, this got harder when we got into the solution space. It’s really important to continue to engage people as you start to work towards solutions.</p>
<h3 id="tailor-communication-even-if-it-means-you-have-to-duplicate">Tailor communication, even if it means you have to duplicate</h3>
<p>It’s harder to communicate when the work might involve change. It’s also difficult to communicate work in progress without people thinking it’s a done deal.</p>
<p>I initially planned to do weeknotes for this piece of work and share them openly, but quite quickly I found that in order to not share anything that might make people worry about their jobs, I had to make them quite banal, and therefore quite boring.</p>
<p>In retrospect, one thing that held me back is my commitment to keeping things open. I prefer to be as transparent as possible, and so I was aiming to do one set of weeknotes that would serve both the exec team and also everyone else.</p>
<p>Unable to produce one size fits all, I did neither, and in fact, I should have done both.</p>
<h3 id="repeat-the-same-information-in-different-ways">Repeat the same information in different ways</h3>
<p>I had a fortnightly meeting with the CPO, CTO and CPTO to update on progress, which was initially a great medium for sharing progress and checking alignment.</p>
<p>This worked great when all three of those execs could show up, but fell apart when one of them was off sick. She missed a step that the other two were apparently on board with, so I continued working on that angle; when she returned she did not agree with the direction. It then took time to both bring her partly up to speed and also to account for what she wanted. Because the meeting was fortnightly, that one missed person in one meeting set the project back six weeks.</p>
<p>I’ve talked above about the importance of clarifying what execs/sponsors are expecting from the work, but you also need multiple ways to communicate.</p>
<p>If you want to communicate something it should be verbal, visual and written, because people take in information in different ways. I had been preparing slides for the updates and then sharing them after the meeting, which I thought was verbal and written. However, since this experience I’ve worked a lot harder on visual representations, on communicating outside meetings as well as inside and on repeating the message in different ways.</p>
<h3 id="dont-start-with-the-biggest-problem-start-by-showing-progress">Don’t start with the biggest problem; start by showing progress</h3>
<p>When we’d completed the internal and external interviews, our prioritisation process was the clear frontrunner among the potential issues to address.</p>
<p>I got very excited about tackling the biggest problem we had and effecting a change that would have a huge impact on our productivity. I also was following the <a href="/jfdi/effective-executive.html">Drucker principle of “First things first, second things not at all”</a> so I threw my energy into addressing that issue.</p>
<p>The mistake I made here was starting with the biggest problem. This issue was one that was stopping all other pieces of work being as successful as they could be, so it seemed to be the highest leverage: but actually, it was not. It was a big problem and would take a lot of energy to solve; a high leverage opportunity is one that has a big impact for a relatively small amount of work.</p>
<p>Meanwhile, because I was focusing on this problem and it was slow going, there was no demonstrable progress on any other part of the work. Part of strategic work is demonstrating that your strategy is sound. You can’t just jump to an ideal end state. So take tangible small steps that you can deliver quickly. This will demonstrate the value of the wider piece of work and earn you the trust to execute the larger pieces of work that may need a leap of faith.</p>
<h3 id="people-resist-by-asking-for-more-data">People resist by asking for more data</h3>
<p>One thing that was new to me was people pushing back on change they don’t want by asking for more data.</p>
<p>Two examples were “What would that look like, can you show some options to help me make a decision?” and “Can you give me some examples of where this has been the root cause of the problem so I can understand the issues better?”</p>
<p>Both of these seemed like reasonable requests, so I went away and diligently pulled together more information. Both times, when I returned with it, they didn’t want the information I’d put together. Asking for more data was a way of resisting the change.</p>
<p>How to address that is a whole other post, but my main learning is, if you’ve got to the point where these questions are being asked, you’ve missed out a step or two on the way of bringing people on the journey, and you need to find a new way to get where you are going.</p>
<h3 id="this-work-takes-a-long-time">This work takes a long time</h3>
<p>As I mentioned above, I genuinely thought I would achieve my strategic goals, have communicated them, and be ready to start the work in three months. Since then, speaking to people, I’ve understood that just the first part of this work can take many months, and the execution of the strategy can take many years, and this isn’t because people are not focusing: it’s that some of this work takes a long time.</p>
<p>This was an incredibly valuable learning for me and helps make sense of some previous situations where I had felt leadership were dragging their feet.</p>
<h3 id="celebrate-what-has-been-done-before-moving-on">Celebrate what has been done before moving on</h3>
<p>This was one of my biggest personal learnings that year. Because of how I’m motivated, I am always looking for the next challenge and opportunity. Sometimes that means I don’t pause to celebrate what I have achieved to date.</p>
<p>That doesn’t help me, because it can mean that I feel like I’ve not achieved much, as I’m always looking forwards at what is left to do. But more importantly, if I don’t celebrate the amazing successes of my team, it doesn’t help them feel their efforts and hard work have been worthwhile; and it doesn’t help executive sponsors see the value of the work.</p>
<p>This is something I’m still working on. You can even see that in this post, where I’ve spent a lot more time talking about how I would improve, than I have talking about the big changes we managed to drive, the engagement of multiple people in making this project successful, and the tangible change of direction of the tech strategy that demonstrably helped teams deliver faster.</p>
<h2 id="this-continues-to-be-very-useful-for-me">This continues to be very useful for me</h2>
<p>If I had to pull out the three lessons that were most valuable to me:</p>
<ul>
<li>Even if you think you know the desired end state, take a smaller chunk and make some tangible steps.</li>
<li>Overcommunicate the goal and your progress towards it.</li>
<li>Focus more on bringing people with you than on getting a perfect answer.</li>
</ul>
<p>This was an incredible learning experience and has really changed the way I approach things.</p>
Scaling People2023-11-26T00:00:00+00:00https://www.annashipman.co.uk/jfdi/scaling-people.html<p>I recently read <a href="https://uk.bookshop.org/p/books/scaling-people-a-tactical-guide-to-company-building-claire-hughes-johnson/7412488">Scaling People</a> by Claire Hughes Johnson, former COO of Stripe. I was first interested in it because it was <a href="https://www.economist.com/business/2023/06/22/scaling-people-is-a-textbook-piece-of-management-writing">reviewed in the Economist (paywall)</a> and then an excellent colleague of mine recommended it. Here are my notes.</p>
<h2 id="this-is-an-enormous-book">This is an enormous book</h2>
<p>It’s about 500 pages, hardcover and structured like a textbook. My colleague <a href="https://www.linkedin.com/in/aaron-lawlor/">Aaron Lawlor</a> suggested the format may be because it is aimed at the business school market.</p>
<p>Having read it, I can imagine there are some bits I might want to refer back to, but overall, it didn’t completely work for me. However there were quite a few interesting parts.</p>
<h2 id="her-central-thesis-is-that-you-need-core-principles-and-processes-that-support-them">Her central thesis is that you need core principles and processes that support them</h2>
<p>“No matter how brilliant a company is, it will not get far, let alone have an impact at scale, without strong management and sound operating systems”. She then she gives guidance on what these might look like.</p>
<p>They start with what she calls “Essential operating principles”, which are the guidelines you use to make decisions and get things done.</p>
<p>Her four are:</p>
<ul>
<li>Build self-awareness to build mutual awareness.</li>
<li>Say the thing you think you cannot say.</li>
<li>Distinguish between management and leadership.</li>
<li>Come back to the operating system.</li>
</ul>
<p>The rest of the book is based on these principles and the “operating system:, i.e. the core processes that she has built around them.</p>
<h2 id="your-core-principles-will-be-different-to-hers">Your core principles will be different to hers</h2>
<p>She acknowledges that your core principles will likely be different. Her position is that you need to work out what yours are in order to drive the operating principles of a company.</p>
<p>Your core principles may not be immediately apparent to you. A good way to start is by working out your most important values. She includes an extract in the book from a <a href="https://nafb.com/sites/default/files/pages/convention/2017%20-%2074th/2017ValuesExercise.pdf">values exercise you can do online (PDF)</a>.</p>
<p>There is an extra step which I could have used a bit more detail on: getting from your core principles to the values of a company where others’ core principles will be different.</p>
<h2 id="say-the-thing-you-think-you-cannot-say">Say the thing you think you cannot say</h2>
<p>This is one of her core principles. There was an interesting thought that has stayed with me:</p>
<div class="quote">In his book Conscious Business, Fred Kofman explains why it's so hard for us to say what we’re thinking. It's because every conversation has three components:
<ul>
<li>The "it": the task being discussed</li>
<li>The "we": the relationship between the people having the discussion</li>
<li>The "I": your personal stance in the conversation</li>
</ul>
Each of these can add difficult unspoken dynamics to the conversation.
</div>
<p>Throughout the book she comes back to this principle, for example around feedback, building trust etc.</p>
<h2 id="when-was-the-last-time-you-leaned-into-the-companys-values-to-make-a-decision">When was the last time you leaned into the company’s values to make a decision?</h2>
<p>The books is really focused on the tools and examples that she built up at Stripe and there is a lot of documentation included.</p>
<p>One really interesting one is the Stripe values document. It’s long, a lot longer than a usual values document, and really paints a picture of what it’s like to work at Stripe.</p>
<p>For me, it made it very clear that it isn’t somewhere I’d want to work, which in itself is useful and does more than a usual values statement does.</p>
<h2 id="things-should-be-uncomfortable">Things should be uncomfortable</h2>
<p>One of her takes on the difference between leadership and management is: “Once you become a great manager, you can get very comfortable. But once you become a true leader, almost every day is uncomfortable.”</p>
<p>“A strategy should hurt”. It should be painful and disappointing, either internally or to your customers. There’s no such thing as a strong strategy that prioritises everything at once.</p>
<p>A key role of leadership is to invest in critical work that no one else will naturally step up to prioritise.</p>
<h2 id="useful-steps-for-setting-goals">Useful steps for setting goals</h2>
<p>The book includes a large extract from this <a href="https://msiliski.medium.com/good-goals-249db6df265d">blog post on good goals</a> by a former product leader at Stripe. The post is worth reading in full.</p>
<p>She also makes some concrete suggestions around the cadence of reviews, for example quarterly business reviews.</p>
<p>I also recently read <a href="https://www.linkedin.com/posts/oliver-at-coda_when-i-was-at-google-okr-planning-took-many-activity-7128426018071072768-nyEw">an interesting post on LinkedIn</a> extolling the virtues of shortening the planning process. For me, the really compelling point there was “because it was so short, it actually also ALLOWED everyone to focus 100% on planning and delay their other work” which is something I and my engineering leadership team have experienced a bit of recently.</p>
<p>This all together has some good, well-thought out ideas on how to run a planning process that works well.</p>
<h2 id="recruitment-doent-end-with-a-successful-hire">Recruitment doen’t end with a successful hire</h2>
<p>She has some great insight into how to improve recruitment. The process doesn’t end with the hire, but with a successful onboarding and a strong connection between the new hire and their company overall.</p>
<p>She talks about getting recognised in a tough market. There are usual ways, like writing blog posts, but Stripe went a lot further than that. <a href="https://techcrunch.com/2012/08/22/stripes-capture-the-flag-2-0-a-hands-on-contest-for-app-developers-to-test-their-security-know-how/">They held several capture the flag competitions</a> which created a strong talent brand and drove leads to the hiring process.</p>
<p>She includes a write-up of the work involved to set that up, and it was a lot. Several weeks, many engineers, a residential week for the core team of engineers in North Dakota for the crunch period. It was a major piece of work, and she doesn’t really dwell on that point. It’s an interesting thought experiment about just how important you think it is to hire the best.</p>
<h2 id="attracting-the-right-candidates">Attracting the right candidates</h2>
<p>Be really clear about the work environment and watch out for language that subtly transmits bias – use tools like <a href="https://textio.com/">Textio</a> to check that.</p>
<p>Make your job ads genuine, not a rosy ad. Be self aware as a company. Knowing who you are helps you hire well.</p>
<p>Think about who is successful at the company. What qualities do they share and what questions can you ask to suss out whether candidates have those qualities?</p>
<p>She also has some great background on how Stripe makes it clear that hiring is everyone’s job. They have some clear principles for all involved: hiring managers, interviews and recruitment, about how to make hiring successful. For example, keep calendar accurate, reply to any candidate question within 48 hours or send on to hiring manager; and on recruitment’s side, be respectful of calendars, shepherd candidates through offsites, etc.</p>
<p>Her advice is that the founder should meet each hire until the the company is about 100 people, because they are the people critical to the company’s trajectory.</p>
<p>She advocates really spending a lot of time on hiring. Talent is everything. Hiring is expensive but it is a critical foundation.</p>
<h2 id="some-useful-interview-questions">Some useful interview questions</h2>
<div class="quote">
“The best interviews suss out how someone:
<ul>
<li>Works with other people</li>
<li>Gets quality work done</li>
<li>Motivates and develops themselves</li>
<li>Has or can develop the expertise needed for the role</li>
<li>Demonstrates leadership and resilience”</li>
</ul>
</div>
<p>It’s good to ask then how their colleagues would describe them and about constructive feedback they’ve received. (I’ve also been asked: how would your reports describe you).</p>
<p>Always make reference calls; at least one reference call should be made by the hiring manager themselves. One question she likes to ask: “Is this person in the top 50 per cent of people you’ve worked with?” Then “top 20? 10, 5?” She says people can be positive, but will usually be honest when asked for data. “If someone only says the candidate is in the top 20 percent of folks they’ve worked with, that’s not a ringing reference.”</p>
<p>She also has some other questions like, where do you see them in three years, when was the last time you didn’t see eye to eye, and what advice do you have for me as a manager as to how I can best support them, which gives insight into their weakness/development areas. This is useful for the hire decision, but is also useful for starting to work with them.</p>
<h2 id="additional-thinking-around-leadership-hires">Additional thinking around leadership hires</h2>
<p>Screen the candidate so that at least two important people are excited by them. Bring in the main stakeholders, even those you think will be negative. Make sure hiring deliberation is a feedback forum not a decision-making forum.</p>
<p>She also stresses the importance of the announcement, which should remind people of the process, celebrate the person you’ve hired and explain why they are right for the company and the role.</p>
<p>It’s key when hiring senior people that you dig into their answers, to make sure they are not just giving you a pre-prepared answer that won’t suit your context. Senior people will be good at interviewing.</p>
<h2 id="detailed-processes-around-onboarding">Detailed processes around onboarding</h2>
<p>This was a section I felt I may refer back to, or pass on to others. A lot of detail on how to onboard, including the “why” of the company and the “how” of company behaviours.</p>
<p>She says company leadership should take part in onboarding, possibly a monthly session where leaders talk about values etc.</p>
<p>If a hiring mistake is made, do a retrospective on your process.</p>
<p>Interesting ideas about processes to track things like internal mobility. Are people moving for career growth, or does it say something about a certain manager or team?</p>
<h2 id="think-beyond-individual-to-team-development">Think beyond individual to team development</h2>
<p>Performance tools are focused on the individual, but it’s teams that get the work done. Investment in teams is not something you can finish; “team development is more like a set of habits you cultivate over time, some performed by the manager and some by team members.”</p>
<p>We should all be learning how to teach and facilitate change management.</p>
<p>At first your work is about building a common team understanding of the type of team you want to be and of the results you want to achieve.</p>
<p>If you are thinking about restructuring a team, she recommends mapping out the structure best suited to accomplish your team’s goals and then write out how you’d communicate it – the narrative about why this is the best structure to support your strategy. Similar to writing a press release before you’ve finished a product, writing a comms plan at this point can expose gaps in your thinking and surface potential objections.</p>
<h2 id="under-delegating-and-over-delegating">Under-delegating and over-delegating</h2>
<p>She has some good examples of where you might be under-delegating or over-delegating to your team.</p>
<p>Under-delegating: you are involved in all the team’s work, you ask to review it all before seen externally. Some clues you may be under-delegating:</p>
<ul>
<li>people come to you with problems but rarely with solutions</li>
<li>decisions can’t be made without you</li>
<li>if you’re on holiday things start falling apart</li>
<li>you feel overwhelmed by your workload and can’t spend time on strategic work</li>
</ul>
<p>Over-delegating: you are good at making employees feel empowered and trusted but you are too far away from the work and may also give people work they are not ready for. Some clues you may be over-delegating:</p>
<ul>
<li>you become aware of projects going off the rails too late</li>
<li>your reports often tell you they feel overwhelmed by their jobs</li>
<li>you can’t say off the top of your head what critical work your team has recently completed, what work is underway and what comes next on the priority list</li>
</ul>
<p>I definitely recognised one or two of the over-delegating examples she gave – sorry previous teams!</p>
<p>When delegating, explain why the work is right for the person, e.g. with reference to their development goals.</p>
<h2 id="i-agree-with-her-ideas-about-meetings">I agree with her ideas about meetings</h2>
<p>Structure, agendas, prep, etc. She has <a href="https://m.youtube.com/watch?v=GIiaFW874q8">given a talk about it</a>.</p>
<p>One new idea to me was checking out. I’ve written before about <a href="/jfdi/meeting-check-in.html">meeting check-ins</a>. We recently did some team training where they said “you haven’t joined a meeting until you’ve spoken” and suggested check-ins for every meeting.</p>
<p>Hughes Johnson’s suggestion is start with maybe just one word on how you are feeling, and then also end the meeting with a checkout, for example, are you happy with the decision we took?</p>
<h2 id="very-useful-suggestions-on-how-to-calibrate-end-of-year-reviews">Very useful suggestions on how to calibrate end of year reviews</h2>
<p>In a larger company, do data analysis, for example average designations for a given group, or promotion percentages between employees of different genders, or remote and non-remote employees.</p>
<p>In a smaller company, a more manual calibation: submit scores ahead of time and look for anomalies. Over time, these meetings become less about reviewing each person in detail and more about gut-checking what each designation means, and finding examples so that every manager can calibrate to what, for example, “meets expectations” means for a certain role and level.</p>
<p>It is important to remind people of the kind of cognitive biases that might surface: beyond unconscious bias, she mentions availability, recency and confirmation biases as examples. Understand and coach the group on how to avoid each one.</p>
<p>Her suggestion of how to actually do the calibration is similar to the process we used when I was at the <a href="https://www.gov.uk/government/organisations/government-digital-service">Government Digital Service</a>. Start at the more junior levels with everyone in the room, and once you start reviewing people who are at the same level as the managers in the room, start excusing the more junior managers.</p>
<p>Do a final review of the data afterwards to check for broad parity. Then roll up to senior leadership to review and check for an expected distribution. It shouldn’t be a forced distribution, but high n counts should have a roughly normal distribution. Senior management should also review that the percentage of employees being promoted feels equitable and in line with compensation budgets.</p>
<h2 id="managing-high-performers">Managing high performers</h2>
<p>She divides high performers into ‘pushers’, who are highly ambitious, can be very critical, set high standards for themselves and others, and are internally motivated, and ‘pullers’, who take on much more work than they should and deliver high-quality output, but they get burned out; they are more externally motivated and have trouble saying no.</p>
<p>She then suggests some ways you can motivate and develop these two kinds of performers.</p>
<p>Pushers: encourage and reward them for the high standards they set; ask them to lead projects to address areas that need attention; frame work as development opportunities; coach them on how to exert influence indirectly.</p>
<p>Pullers: help them prioritise and set boundaries; encourage and reward them in private and publically; before they take work on encourage them to ask themseves: am I the most qualified person to do this? Is this the most important thing I could be doing? What would I have to let go in order to do this? Coach them on how to say no.</p>
<h2 id="do-the-best-for-your-reports">Do the best for your reports</h2>
<p>Aniticipate when the work will become boring and work with your reports to think about where they might want to focus in six months. Include them in your succession planning. Think laterally about opportunities that might give them a chance to grow in the ways they want to, even if they aren’t their logical next step.</p>
<p>Finally, let go. If you can’t find interesting opportunites for a high performer on your team, don’t hold onto them for too long – land them a job in another part of the organisation, or open up your network and help them get another job; play the long game, you’ll have earned their loyalty and trust.</p>
<p>She also suggests pre-exit interviews – ask people: if you were going to leave, what would be the reason? Ask 3-5 people, or maybe the top 10% of talent.</p>
<h2 id="managing-good-performers">Managing good performers</h2>
<p>Don’t just spend time with your high performers, also think about people who are not currently high performers but have a lot of potential. She says she obsessess over how to unlock the upside.</p>
<p>For example, someone who always has a completely different perspective on a problem can sometimes feel very disruptive, but they can also engineer the breakthrough the team are seeking. She gives an example of when she was at Google. Someone kept suggesting peer-to-peer support as a solution to their customer support backlog. Eventually she told them “find an external example and write up a way for us to test the concept”. The test was very successful, users were getting help immediately, and it led to a building a new support approach that is still in use.</p>
<p>Finally, she notes medium performers are sources of stability and are often culture carriers. This ties into something we talked about on my <a href="https://www.iod.com/professional-development/open-courses/leadership-for-directors/">IoD leadership course</a> (yet to be written up!) that the ‘medium’ performers, who are competent, consistent, and productive, but aren’t your highest performers, are actually the pillars of the company.</p>
<h2 id="managing-poor-performers">Managing poor performers</h2>
<p>She also goes into a lot of detail about managing poor performers.</p>
<p>One of the things she does in this book is suggest different ways to say things:</p>
<div class="quote">
"So rather than "The team is performing terribly," you might say, "My impression is that the team isn't communicating well right now, and I think deadlines are being missed as a result. Do you agree, or could it be something else?".
</div>
<p>I found this a bit grating, and there was a lot of it in the section on managing poor performance. This could be useful for people who don’t have experience here or want advice on exactly what words to use, but it didn’t work for me.</p>
<p>One thing to note is that the content on managing out is definitely not useful if you are not based in the US.</p>
<h2 id="who-is-this-book-for">Who is this book for?</h2>
<p>This book contained a lot of great ideas and pulled together a lot of useful resources, but as a whole, I found it tiring, rather than inspiring. I am not sure if that was the style, or just that I am not the target audience. It was definitely management rather than leadership.</p>
<p>It’s also very long. Hughes Johnson says it’s not necessarily to be read cover-to-cover but instead pick it up and go to the page of what’s on your mind. However, for that to be the case, the contents pages and index need to be a lot better.</p>
<p>If you are in a company that doesn’t have good processes, particularly around people management, then this definitely has a lot of useful content and good ideas, but it wasn’t a strong recommend from me.</p>
Analysing, Deciding Doing: How to develop and execute an effective strategy2023-09-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/analysing-deciding-doing.html<p>In June 2023, I delivered a talk at <a href="https://turingfest.com/">Turing Fest</a> on how to create a strategy. This talk is the one I would have liked to have seen when writing my first strategy: how you actually do strategy.</p>
<h2 id="watch-the-talk">Watch the talk!</h2>
<div class="embedded">
<script src="https://fast.wistia.com/embed/medias/3ixjibkh5x.jsonp" async=""></script><script src="https://fast.wistia.com/assets/external/E-v1.js" async=""></script><div class="wistia_responsive_padding" style="padding:56.25% 0 0 0;position:relative;"><div class="wistia_responsive_wrapper" style="height:100%;left:0;position:absolute;top:0;width:100%;"><div class="wistia_embed wistia_async_3ixjibkh5x seo=true videoFoam=true" style="height:100%;position:relative;width:100%"><div class="wistia_swatch" style="height:100%;left:0;opacity:0;overflow:hidden;position:absolute;top:0;transition:opacity 200ms;width:100%;"><img src="https://fast.wistia.com/embed/medias/3ixjibkh5x/swatch" style="filter:blur(5px);height:100%;object-fit:contain;width:100%;" alt="" aria-hidden="true" onload="this.parentNode.style.opacity=1;" /></div></div></div></div>
</div>
<p>The slides are here:</p>
<iframe src="https://www.slideshare.net/slideshow/embed_code/key/xTSTaaoOrur50J?startSlide=1" width="597" height="486" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px;max-width: 100%;" allowfullscreen=""></iframe>
<h2 id="summary-of-the-talk-models-and-techniques-to-help-you-create-your-strategy">Summary of the talk: models and techniques to help you create your strategy</h2>
<p>In the talk, I go through some specific models, with examples.</p>
<ul>
<li>Analysing the situation: how to reach a diagnosis, including models like PESTLE, S-curves, Five Forces, VRIN, and value chain analysis.</li>
<li>Making a strategic decision: how to make a strategic choice, including Three Horizons, Ansoff matrix, Blue Ocean strategies, and weighting and scoring.</li>
<li>Making it happen: how to implement your strategy, including communication, managing change and showing progress.</li>
</ul>
<p>These tools, techniques and actionable advice will set you up to be able to create and deliver an effective strategy.</p>
Strategic technology leadership: video2023-09-03T00:00:00+00:00https://www.annashipman.co.uk/jfdi/strategy-fireside-video.html<p>Earlier this year, I keynoted the second day of <a href="https://conference.ctocraft.com/london-2023/">CTO Craft Con: The Strategic CTO</a>, and the video is now available:</p>
<div class="embedded">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/g0fnRq2KAho" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
</div>
<p>You can also read more about the discussion in <a href="/jfdi/strategy-fireside-chat.html">my write-up</a>.</p>
Strategic technology leadership: fireside chat2023-07-11T00:00:00+00:00https://www.annashipman.co.uk/jfdi/strategy-fireside-chat.html<p>I keynoted the second day of <a href="https://conference.ctocraft.com/london-2023/">CTO Craft Con: The Strategic CTO</a>. The topic was strategic technology leadership. This is a write-up; you can <a href="/jfdi/strategy-fireside-video.html">watch the video here</a>.</p>
<h2 id="it-was-a-fireside-chat">It was a fireside chat</h2>
<p>I was in conversation with the excellent <a href="https://www.linkedin.com/in/emmahopkinsonspark/">Emma Hopkinson-Spark</a>.</p>
<p>Emma kicked it off by asking what strategy actually is, and how it’s different from roadmaps, initiatives, tactics etc. That gave me an opportunity to define strategy as diagnosis, vision and a plan, and recommend the excellent <a href="/jfdi/good-strategy-bad-strategy.html">Good Strategy, Bad Strategy</a>.</p>
<p>We talked about communication. The most important aspect of strategy is communication: the point of a strategy is to execute on it, so people need to understand what they need to do.</p>
<p>It needs to be simple enough for people to understand; communication needs to be verbal, visual and written because people absorb information in different ways, and you need to keep repeating it, far more than you might think.</p>
<h2 id="you-must-carve-out-time-for-strategy">You must carve out time for strategy</h2>
<p>There were also questions from the audience. One was “How do you break out of the delivery whirlwind to set your strategy?” and I was unequivocal on this: as CTO it is your job to set and lead on that strategy, so you have to make time. That could be blocking out time, setting meetings with people to work on it, Away Days, or whatever works for you.</p>
<p>When you make time for strategy, the reactive demands decrease because you have more systemic approaches. I also taked about <a href="/jfdi/prioritising-in-tech-leadership.html">rocks, pebbles and sand</a>; strategy must be one of the rocks.</p>
<p>It can be really difficult to carve time out to begin with, but once you have, you will start to see the benefits.</p>
<h2 id="i-favour-transparency">I favour transparency</h2>
<p>Another question was: “what’s the best approach for course correction and how do you tell the difficult story to the board when it’s a drastic change?”</p>
<p>Talking to the board, you should operate on the principle of “no surprises”, so when you see things heading off track, have the conversation as soon as you can.</p>
<p>This led into a discussion how to have that conversation. Emma asked how you change your style if you are communicating upwards, to the board; and one thing I shared was that it’s a good idea to create artefacts (diagrams, documents, progress indicators) and use the same artefacts for everyone: for the board, for your peers, and for your reports, so that everyone knows you are talking about the same thing.</p>
<h2 id="people-are-incentivised-by-working-on-important-things">People are incentivised by working on important things</h2>
<p>Another question from the audience was “how do you directly incentivise people at all levels around a common strategy?”</p>
<p>This was an interesting question because I don’t think of it like that. Part of the work of strategy is inspiring people with the vision. When you are working on the plan, the more you can involve people, the more bought in they will be.</p>
<p>I don’t think about separately incentivising people, because I find they are motivated by working towards the vision they’ve (hopefully) bought into, with some actions they’ve helped figure out. People like to be involved in delivering something good.</p>
<h2 id="how-to-get-started">How to get started</h2>
<p>Emma then asked for suggestions on how to actually get started on strategy work.</p>
<p>I said:</p>
<ul>
<li>carve out the time</li>
<li>be really clear on the diagnosis; you need to know where you are starting from</li>
<li>understand where you’re headed; what is different about the world when you’ve achieved your vision?</li>
<li>communicate. Is it simple, does it have a narrative, can you explain it to the board, your peers and your team?</li>
<li>keep getting feedback throughout the process</li>
</ul>
<h2 id="we-discussed-a-lot-of-other-topics">We discussed a lot of other topics</h2>
<p>There were a lot of other great questions, like how OKRs and strategy can work together, aspiring to have an elevator pitch for your strategy, and thinking about strategy mistakes I’ve made.</p>
<p>You can <a href="/jfdi/strategy-fireside-video.html">watch the full conversation here</a>.</p>
The 7 Habits of Highly Effective People2023-06-26T00:00:00+00:00https://www.annashipman.co.uk/jfdi/7-habits.html<p>I have recently read <a href="https://uk.bookshop.org/p/books/the-7-habits-of-highly-effective-people-revised-and-updated-30th-anniversary-edition-stephen-r-covey/3555997">The 7 Habits of Highly Effective People</a>. It was not what I’d expected, and very good. Here are my notes.</p>
<h2 id="a-lot-of-it-seems-familiar">A lot of it seems familiar</h2>
<p>I resisted reading this for a long time because it seemed the most self-helpy of the business books on my list.</p>
<p>However, it’s referenced all the time. Some of the principles I already try to follow, for example ‘put first things first’ and ‘begin with the end in mind’, and some I definitely feel I could do better, for example ‘seek first to understand, then be understood’, so I thought I would go back to the source material.</p>
<h2 id="it-has-a-really-intense-beginning">It has a really intense beginning</h2>
<p>The first few chapters of the book are really about how you are as a person. “Inside out” – the book is about how to be a decent person, genuinely, rather than how to do some things that make you effective.</p>
<p>This is not at all what is implied by the title; it is indeed more self-helpy than your standard business book.</p>
<p>The author, Stephen R. Covey, talks about the personality ethic (i.e. techniques) vs the character ethic (i.e. things like integrity, humility). His contention is that it’s only the latter that will actually make a difference. If you follow techniques but they don’t come “inside out” then people will be able to tell.</p>
<p>“We see the world not as it is but as we are”. He says character is a composite of habits.</p>
<h3 id="how-the-7-habits-fit-together">How the 7 habits fit together</h3>
<p>He then talks about maturity; initially you are dependent, then you are independent (the first three habits), then interdependent (i.e. part of a bigger ecosystem – the second three habits). The last habit is continuous improvement – renewal (“Sharpen the saw”).</p>
<p>He calls the first three “private victory” and the second three “public victory” and the point is that “inside out” means you have to master private victory before you can master public victory.</p>
<p>He says though, ironically, you will find that as you care less about what others think of you, you will care more about they think of themselves and their world, which includes their relationship with you.</p>
<h3 id="ppc-balance">P/PC balance</h3>
<p>He then talks about maintenance. He calls this “P” and “PC” balance, where P is production, and PC is production capability. You need to look after the goose that lays the golden egg. “Always treat your employees exactly as you want them to treat your best customers”. PC is treating employees as volunteers because that’s what they are; you pay for their labour but they volunteer the best bit: their hearts and minds.</p>
<p>Interestingly he suggests reading the book as if you were going to teach it. Which I always do, by making notes, which I know I will share if I rate the book.</p>
<h2 id="habit-1-be-proactive">Habit 1: Be proactive</h2>
<p>We are free to choose how we react and respond – not strictly programmed by upbringing etc. “Between stimulus and response is our greatest power – the freedom to choose”.</p>
<p>People wait for something to happen or someone to take care of them. But the people who end up with the good jobs are the ones who are solutions to problems, not problems themselves, who seize the initiative to do whatever is necessary, consistent with correct principles, to get the job done.</p>
<p>He said that whenever their kids came to him or his wife with problems, they would say “use your R and I!” (Resourcefulness and initiative).</p>
<p>He talks about proactive vs reactive language. For example, “I don’t have time”; it’s the transfer of responsibility to circumstances, when actually it’s your responsibility. You have chosen to prioritise other things.</p>
<p>Some really interesting examples of alternatives, for example instead of the reactive “I must”, you can use the proactive: “I prefer”.</p>
<p>And the real problem with reactive language is that you yourself hear what you say and it becomes a self-fulfilling prophecy – you feel you are not in control of things that you actually are in control of.</p>
<h3 id="circle-of-concern-and-of-influence">Circle of concern and of influence</h3>
<p><img src="/img/circle_concern_influence.png" alt="A circle inside another; outer circle labelled 'circle of concern', inner circle labelled 'circle of influence'" /></p>
<p>The circle of concern are the things you are concerned about: problems at work, health, children, global warming, etc. The circle of influence, (usually) smaller than that, is where you can actually have an impact.</p>
<p>If you find you are focusing much of your energy on the bit that is outside of the circle of influence, then that tells you that you are more reactive than proactive – you expend time and energy worrying about things that you cannot change. If instead you focus in the circle of influence, this will have a positive effect, and will actually expand your circle of influence.</p>
<p>This also echoes something my coach has said to me: where you focus your attention is where you will find more. So if you worry about problems, you will find more problems. If you think a lot about opportunities, you will find more opportunities.</p>
<p>Problems are direct, indirect or no control. Direct: work on ourselves. Indirect, eg involving other people’s behaviour: work on our influencing (the second 3 habits). No control: have <a href="https://www.alcoholics-anonymous.org.uk/Members/Fellowship-Magazines/SHARE-Magazine/December-2019/The-Serenity-Prayer-and-Me">“the serenity to accept the things that cannot be changed”</a>.</p>
<p>An example of something we have no control over is mistakes we’ve made in the past. Our last mistakes are out there, we can’t correct them. But our response to a mistake affects the quality of the next moment.</p>
<h3 id="this-struck-home-for-me">This struck home for me</h3>
<p>Some of this was hard to read and definitely hit a nerve. I think of myself as proactive, but I could see myself in some of the examples. It feels like a hard lesson but a worthwhile one, and definitely felt like the right time to be reading this for me.</p>
<h2 id="habit-2-begin-with-the-end-in-mind">Habit 2: Begin with the end in mind</h2>
<p>I had thought this was like <a href="https://www.allthingsdistributed.com/2006/11/working_backwards.html">working backwards</a> but it is bigger than that. It actually begins by imagining yourself at your funeral. What would you want people to say about you? “By keeping that end clearly in mind, you can make certain that whatever you do on any particular day does not violate the criteria you have defined as supremely important, and that each day of your life contributes in a meaningful way to the version you have of your life as a whole.”</p>
<p>Figure out your values then begin each day with them in mind, then you can face the challenges and make your decision based on those values, rather than react to the emotion or the situation.</p>
<p>He suggests writing a personal mission statement, and there is actually <a href="https://msb.franklincovey.com/">a tool for doing this on the FranklinCovey site</a>. Your mission statement is for all your roles: professional, personal, in your community, etc.</p>
<p>He then goes on to talk about having a family mission statement! And then onto a company mission statement, which should be written by everyone, not just a few top execs. “No involvement, no commitment.” And then you have to actually use it as your frame of reference in making decisions.</p>
<p>He paraphrases Peter Drucker, saying management is doing things right, leadership is doing the right things. (Incidentally, this is not actually a Peter Drucker quote; <a href="https://www.drucker.institute/did-peter-drucker-say-that/">The Drucker Institute</a> calls this the “the Moby Dick of Drucker misquotations”). Creating a mission statement/thinking about the principles is where you start.</p>
<h2 id="habit-3-first-things-first">Habit 3: First things first</h2>
<p>This is about effective management of yourself once you’ve worked out what it is you want to do. He digs a lot into the Eisenhower matrix (urgent/important) and it’s a useful discussion.</p>
<p>There is an exercise (in appendix) about using the matrix; you are Director of Marketing for a major pharmaceutical firm and you start the day with 10 items on your list. You then think about them with a “quadrant 2” mindset (i.e. important but not urgent – the place where you should be spending most of your time) and then he goes through each item with suggestions, and it was totally eye-opening.</p>
<p>I am good at prioritising and using my time and energy where it is most strategic and impactful, and good at thinking about what and how to delegate, but this discussion was really next level.</p>
<p>One small example: one of the items was catching up on reading medical journals. I smugly noted to myself that this is definitely a quadrant 2 activity I should make time for, but his suggestion was: set up a systematic approach among your team, have each keep up with different journals and teach the essence of what they learned at the next staff meeting, which is so much better because then <em>everybody</em> learns, not just you.</p>
<p>Another example was that you’ve heard rumours of quality control issues. If it has a chronic or persistent dimension to it “you could delegate to others the careful analysis of that chronic problem with instructions to bring to you a recommendation, or perhaps simply to implement what they come up with and inform you of the results”.</p>
<p>If you are in a small business or you are thinking about activities within your family you can’t delegate, but you can still expand your circle of influence to be more quadrant 2.</p>
<h3 id="if-you-cant-focus-on-your-priorities-have-you-really-internalised-them">If you can’t focus on your priorities, have you really internalised them?</h3>
<p>If you find you can’t focus on your priorities maybe it’s because you’ve not really internalised your principles, your mission statement. He describes this as focusing on leaves not roots.</p>
<p>It’s very hard to say no to something urgent if you don’t have a bigger yes burning inside. When you have, you then will have sufficient willpower to say no with a genuine smile to the unimportant.</p>
<h3 id="primary-focus-on-relationships-and-results-secondary-focus-on-time">Primary focus on relationships and results, secondary focus on time</h3>
<p>His suggestion is to set weekly goals for each of your roles – e.g. parent, spouse, manager of person x, manager of project y, community activity – and schedule time in for each of them. “The key is not to prioritise what’s on your schedule, but to schedule your priorities.”</p>
<p>However, you can’t think efficiency with people – effectiveness with people, efficiency with things. He gives the example of getting frustrated with not getting stuff done because you are spending time with little kids. But “frustration is a function of our expectations, and our frustrations are often a reflection of the social mirror rather than our own values and priorities”. If you have habit 2 in your heart, you can subordinate your schedule to those values with integrity.</p>
<h2 id="the-second-set-of-habits-public-victoryinterdependence">The second set of habits: public victory/interdependence</h2>
<p>The second set of habits are working with other people, but he says you need to work on the first three habits first. You can’t jump into building effective relationships if you haven’t done the work on yourself first.</p>
<p>“The most important ingredient we put into any relationship is not what we say or what we do, but what we are”. He talks about making deposits or withdrawals into an “emotional bank account”. This is about building trust.</p>
<p>Understand people and give them things they want, not what you would want. Pay attention to small tokens of affection. Keep commitments. Do the sometimes uncomfortable work of clarifying expectations.</p>
<p>Show personal integrity. One of the ways to do this is to show loyalty to people who are not present. In doing so you build the trust of those who are present. Integrity is treating everyone by the same set of principles. They may not appreciate the honest confrontation at first, but in the long run people will trust you if you are honest and kind.</p>
<p>Apologise quickly if you break these. People will forgive mistakes because they are of the mind, judgment. Will less easily forgive mistakes of the heart, ill intention, bad motive.</p>
<p>Problems are opportunities to build a relationship.</p>
<h2 id="habit-4-think-winwin">Habit 4: Think win/win</h2>
<p>This is based on the paradigm that there is plenty for everybody, that one person’s success is not achieved at the expense of the success of others. Abundance. Looking for all to succeed and share in the spoils.</p>
<p>Win/win is where both sides in a negotiation feel like they have won. He goes through some alternatives to win/win: win/lose, lose/lose, win and not care about anyone else. In some cases, the other ones are useful, but in general if it’s not win/win, then it’s lose/lose over the long run. For example, if you win in a negotiation with a supplier, will they feel as happy working for you again in the future? Will they do their best work now?</p>
<p>Sometimes things can appear to be win/win, but on examination they actually are lose/win (he gives some examples). He suggests staying longer in the communication process, listening more, continuing in the win/win spirit until a solution is reached that both sides feel good about. And that solution would’ve been “synergistic”, i.e. greater than the sum of its parts. Spending time working together like this probably leads to something neither of you would’ve thought of on your own.</p>
<p>He suggests win/win, or no deal. That is, if we can’t come to a solution where both side feels like they’ve won, then we should amicably walk away. There may be another deal in future. Any deal that feels like less than win/win will have impacts on the long-term relationship.</p>
<p>For example, if family members can’t agree on a video that everyone will enjoy, they can instead decide to do something else – no deal – rather than have some enjoy the evening at the expense of others. “Provides tremendous emotional freedom in the family relationship.”</p>
<p>Thinking win/win involves the unique human endowments: self-awareness, imagination, conscience, independent will; mutual learning, mutual influence, and mutual benefits. It takes great courage as well as consideration, especially if we are interacting with those who are deeply win/lose.</p>
<h3 id="thinking-winwin-takes-maturity">Thinking win/win takes maturity</h3>
<p>It is incredibly hard to think win/win when others are win/lose. It requires a lot from you. This accords with my experience trying to apply some of this when reading the book.</p>
<p>It takes a balance of courage and consideration. Courage to get golden egg, consideration of the long-term welfare of the other stakeholders. “The basic task of leadership is to increase the standard of living and the quality of life for all stakeholders”.</p>
<p><img src="/img/consideration_courage.png" alt="Diagram showing low-high consideration and low-high courage; high of both leads to win/win" /></p>
<p>When you are dealing with someone strong win/lose, the relationship is still the key. “Make deposits into the emotional bank account through genuine courtesy, respect and appreciation for that person and for the other point of view”. Listen more. “Keep hammering it out until the other person begins to realise that you genuinely want the resolution to be a real win for both of you.”</p>
<p>The more genuine you are, and the more committed to win/win, the more powerful your influence will be with that other person. “This is the real test of interpersonal leadership. It goes beyond <em>transactional</em> leadership into <em>transformational</em> leadership.”</p>
<p>He also suggests this can help you set up home responsibilities that eliminate constant nagging and enable parents to do the things only they can do.</p>
<h3 id="how-to-do-it">How to do it</h3>
<p>Win/win cannot survive in an environment of competition. The systems have to support it. The training system, the communication system, the budgeting, compensation, etc.</p>
<p>For getting to win/win he recommends <a href="https://uk.bookshop.org/p/books/getting-to-yes-negotiating-an-agreement-without-giving-in-roger-fisher/3126697">Getting to Yes</a>. Principled negotiation. Separate the person from the problem. Focus on interests and not positions. Invent options for mutual gain. Insist on objective criteria.</p>
<p>His version of that is:</p>
<ol>
<li>See the problem from their point of view.</li>
<li>Identify the key issue and concerns (not positions).</li>
<li>Determine what results would constitute a fully acceptable solution.</li>
<li>Identify new possible options to achieve those results.</li>
</ol>
<p>But the point is – the end and means are the same. You can only achieve win/win results with win/win process.</p>
<h2 id="habit-5-seek-first-to-understand-then-be-understood">Habit 5: Seek first to understand, then be understood</h2>
<p>He talks about empathetic listening. Not just attentive listening but really trying to understand. Focused on receiving the deep communication of another human soul. His suggestion about how to do this is both to rephrase the content and also reflect the feeling “you’re feeling frustrated about school”.</p>
<p>It has to come from a sincere desire to understand, not trying to manipulate. He works through some conversations with suggestions.</p>
<p>When you are then seeking to be understood, he suggests following Aristotle’s suggestions around rhetoric (the art of persuasion), particularly <em>ethos</em>, <em>pathos</em>, and <em>logos</em> in that order. i.e. your personal credibility; then the feeling, being in alignment with the emotional thrust of the other person’s communication; and then finally the logical part of the argument. Most of us lead out with the third part and attempt to convince with just that.</p>
<p>This really chimed with some advice from my coach when I was preparing a pitch for a piece of work; the audience will feel, then think, then react. They need to feel confident about themselves; trusted, valued, appreciated, acknowledged. How they feel about themselves will affect how they feel about what you’re suggesting, especially if you are trying to create change.</p>
<p>Describe the alternative they are in favour of better than they can themselves. “Let me see if I first understand what your objectives are, and what your concerns are about this presentation and my recommendation”.</p>
<p>Habit 5 lifts you to greater accuracy, greater integrity in your presentations. When you can present your own ideas in the context of a deep understanding of other people’s concerns you significantly increase the credibility of your own ideas, and “what you’re presenting may even be different from what you had originally thought, because in your effort to understand, you learned”.</p>
<p>He suggests doing this in as many contexts as possible. For example, set up stakeholder information systems to get honest, accurate feedback at every level. Make the human element as important as the financial or technical element. You save lots of time when you tap into the human resources of a business at every level. When you listen, you learn.</p>
<p>Habit 5 also increases your circle of influence because you really listen, so you become influenceable, and being influenceable is the key to influencing others.</p>
<p>“When we really, deeply understand each other, we open the door to creative solutions and third alternatives. Our differences are no longer stumbling blocks to communication and progress. Instead they become the stepping-stones to synergy”.</p>
<h2 id="habit-6-synergize">Habit 6: Synergize</h2>
<p>Distressingly, while I was reading this book, I found myself using the word “synergy” in a conversation. The principle of it is good – when combined efforts produce an effect greater than the sum of its parts – but the word itself is so ickily business jargon it was embarrassing.</p>
<p>However, if you can get past that, the concept itself is the holy grail; definitely worth striving for.</p>
<p>It’s about valuing difference. The difference is where the value comes from.</p>
<p>It is creative. When you “communicate synergistically” you are opening yourself to new possibilities. You are being comfortable with ambiguity. It may seem the opposite of habit 2, begin with the end in mind, but it’s actually fulfilling it. You are not sure how things will work out, but you have a sense of excitement and you believe the outcome will be significantly better than it was before, and that is the end you have in mind.</p>
<p>Relationship building is an important investment. Attitude of “if a person of your intelligence and competence and commitment disagrees with me then there must be something to your disagreement that I don’t understand, that I need to understand”.</p>
<p>Respectful communication is a mature approach, but it does not open creative possibilities. Both sides give and take, it may be high trust, honest and genuine, but it is not creative.</p>
<p>It is about what he calls “fishing for the third alternative”. He gives a very nice example about a disagreement between a woman and her husband about how to use their holiday time. As I was reading it, I really could not think of how they could come up with a third way; their positions both seemed completely valid and mutually exclusive. But he takes you through how together they create some possible solutions that will work for both.</p>
<p>He talks about recognising one’s own limitations. We might think “I see the bigger picture! I am objective. Everyone else is buried in the minutiae”. But the person who is truly effective has the humility to recognise her own perceptual limitations, and to appreciate the rich resource available through interaction with the hearts and minds of other people.</p>
<p>Value the difference. Good! You see it differently! Tell me what you see. If two people have the same opinion, one is unnecessary.</p>
<p>You can also value the difference within yourself. You yourself have an analytical side and a creative side. “You can be synergistic within yourself even in the midst of a very adversarial environment. You don’t have to take insults personally. You can sidestep negative energy; you can look for the good in others and utilise that good, as different as it may be, to improve your point of view and to enlarge your perspective.” For example, don’t focus on your boss’s weaknesses, don’t complain. Everyone has weaknesses – you yourself do – so work with their strengths and complement their weaknesses.</p>
<p>You can use your courage to express your feelings etc in a way that will encourage other people to also be open. When someone disagrees with you, you can affirm them, and seek to understand. There is almost always a middle way. And if you work win/win, and really seek to understand, you can usually find a solution that will be better for everyone concerned.</p>
<h2 id="habit-7-sharpen-the-saw">Habit 7: Sharpen the saw</h2>
<p>“Habit 7 is Personal PC”, i.e. it’s looking after yourself. Physically, socially, emotionally, mentally and spiritually. Spend time on all of these things. He recommends his own process, “daily private victory” which is an hour a day in renewal of the physical and mental dimensions, e.g. exercise; reading literature; prayer if that’s your thing, etc.</p>
<p>He give an analogy with exercise and emotional muscles. When you exercise your patience beyond your past limits, the emotional fibre is broken, nature overcompensates and the next time the fibre is stronger. You can practice in day to day interactions. It’s hard to disagree on fundamental things, but you can practice and improve.</p>
<p>If you settle the inward battles, you feel a sense of peace, a sense of knowing what you are about. Personal mission statement. Then public victories will follow naturally. Security also comes from stepping out of your own frame of reference without giving it up. Also from meaningful projects, helping others.</p>
<p>Growing other people by believing in them, affirming them, encouraging them to be proactive. Reflecting to them their unseen potential. See them in fresh new ways each time we are with them. Treat them how they could be.</p>
<h2 id="i-recommend-this-book">I recommend this book</h2>
<p>I found this incredibly useful. While I was still reading it, I had a conversation with someone at work that I’d had a difficult relationship with and we were at opposite ends of an issue. Rather than coming in with my perspective I listened first to hers and reflected it back, and thought about her perspective; and shared info with her about how I’d come to mine, and together we did actually come up with a new proposal that was better, and it was a more pleasant conversation.</p>
<p>It is a big book and can be heavy going, but it is worth the effort. There is a lot to it, and re-reading parts for this write-up reminded me there is a lot more to take from it.</p>
<p>It’s a classic for a reason, and I recommend you read it.</p>
Software Architecture Hour with Neal Ford2023-05-20T00:00:00+00:00https://www.annashipman.co.uk/jfdi/architecture-hour.html<p>A few weeks ago I joined Neal Ford on his <a href="https://learning.oreilly.com/live-events/software-architecture-hour-technical-strategy-with-anna-shipman/0636920090704/0636920090703/">Software Architecture Hour</a> to talk about technical strategy. I really enjoyed this conversation; we covered a really broad range of topics, mostly driven by audience questions. We talked about building strategy and selling it, how to do architecture across big groups, how governance can not be a dirty word and more.</p>
<p>If you have an O’Reilly account, you can <a href="https://learning.oreilly.com/videos/software-architecture-hour/0636920937104/0636920937104-video345687/">watch the video here</a>.</p>
What next for No next Next?2022-12-12T00:00:00+00:00https://www.annashipman.co.uk/jfdi/what-next-for-no-next-next.html<p>Just over three years ago, I wrote about the tech strategy for Customer Products, <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">No next Next</a>. In this post I talk about how we’ve implemented our strategy, what the outcomes are, and what we are going to do next.</p>
<h2 id="next-brought-a-faster-coherent-user-experience">Next brought a faster, coherent user experience</h2>
<p>“Next” replaced our previous FT.com site, which was powered by a monolith, called “Falcon”. Falcon could only be deployed monthly, out of hours, and Next, powered by a microservices architecture, brought huge performance and productivity gains.</p>
<p>Falcon also didn’t have one team in charge of the whole; different parts of the business owned different parts of the site. Next brought all of the website estate under one leadership team, meaning that product, UX and design work together to form a coherent user experience, and the tech is governed in one place making it easier to improve it, maintain it and keep it secure.</p>
<h2 id="fighting-entropy-no-next-next">Fighting entropy: No next Next</h2>
<p>It cost £10mn and took us two years to build Next. We wanted to make sure we kept that drive and focus on continuous improvement, rather than letting the tech drift so far off track that we had to throw it away and spend another £10mn and two years building another Next: our tech strategy is No next Next.</p>
<p>In any case, there would be no point in throwing it away and building another stack, because entropy is inevitable. The second law of thermodynamics states that everything tends towards entropy. Unless you actively work towards order, over time your system will become messy, more complex, and ultimately unmanageable.</p>
<p>What No next Next means is a focus on sustainability, on supportability and on simplicity. Maintaining a focus on order, continuous improvement, and active replacement of parts of the codebase that have become too complex. And accepting that entropy is inevitable and finding ways to handle it.</p>
<p><img src="/img/disorder.png" alt="Dots in neat rows on the left leading to dots all over the place on the right, showing things tending towards disorder" /></p>
<h2 id="giving-teams-the-support-to-be-great">Giving teams the support to be great</h2>
<p>The Customer Products team are an incredibly smart, talented and kind group of people, and all of the ideas and execution for how we move forward on our tech strategy came from the team. Together, we realised we needed to make the space for those ideas and execution to flourish.</p>
<p>One of the early changes we made was moving to a <a href="https://medium.com/ft-product-technology/unlocking-value-with-durable-teams-a70efb435a19">stable team structure</a>, rather than forming teams around initiatives. When teams are long-lasting, it is easier to have a clear strategic vision and work towards it, and it also means teams have a chance to learn how to work together to perform really well and become more than a sum of their parts.</p>
<p>As part of that, we addressed the problem that over 80% of our repos didn’t have a clear technical owner. That didn’t mean that there wasn’t someone who owned each system, but we had no way of knowing which ones were owned and which were not. Unowned tech is an operational risk you don’t know about yet: you don’t want to find out in a production incident that actually, the only people who knew about that system have left the company.</p>
<p>With the move to durable teams, we also moved to full technical ownership. This did have challenges, as some teams found they owned a lot of systems, some of which they knew nothing about, and over time we are working on managing this, including sunsetting some tech, and working out where ownership may actually be in another group.</p>
<p>We also worked on improving our recruitment process, for example <a href="https://medium.com/ft-product-technology/making-recruitment-fair-for-people-of-colour-66a3ad907a7d">making it fairer for people of colour</a> so that we can continue to build great teams.</p>
<h2 id="replacing-parts-of-the-codebase-that-have-become-too-complex">Replacing parts of the codebase that have become too complex</h2>
<p>We <a href="https://medium.com/ft-product-technology/achievement-unlocked-6edbc0b44ddd">rebuilt a critical part of our website</a>, called n-ui, on which all of our user-facing apps depended. N-ui did a lot of work, including building and loading client-side code, configuring and loading templates, as well as tracking, ads configuration and a lot more. It was difficult to maintain, not well understood by the current team, and tightly coupled to technical decisions made at the beginning of building the new FT.com. <a href="https://medium.com/ft-product-technology/designing-a-sustainable-front-end-toolset-for-ft-com-f37c59d27eeb">We replaced it with a set of loosely coupled packages called Page Kit</a> that have been much easier to maintain, and as an additional benefit, replacing n-ui with Page Kit made FT.com faster, halving our Time To Interactive (TTI) metric.</p>
<p>We are now working on <a href="https://medium.com/ft-product-technology/unspaghettiing-ft-coms-content-pipeline-be1421a434cb">rationalising some of our content APIs</a> which, over time, have grown organically to be a bit like spaghetti.</p>
<p><img src="/img/spaghetti_apis.png" alt="Various systems connected by lines denoting APIs that overlap each other and look complex" /></p>
<p>We’ve also developed <a href="https://github.com/Financial-Times/dotcom-tool-kit">FT.com Tool Kit</a> to make it easier for teams across FT.com to install the tooling they need, and helps us keep our codebase consistent and maintainable.</p>
<p>As well as working on the less visible parts of the tech estate that slow down performance and developer productivity, we have also worked on simplifying and improving how we deliver features to make it easier for our colleagues elsewhere in the business to work with us. For example, we have recently <a href="https://medium.com/ft-product-technology/read-all-about-it-rebuilding-the-ft-com-newsletters-page-4a63cb16bf43">rebuilt the newsletter page</a>, and we are about to start work on improving the cancellation journey.</p>
<h2 id="improving-our-supportability">Improving our supportability</h2>
<p>When I joined, we only had 5 people on the out of hours rota, and all of those people were people who’d worked on the original build of Next; new people didn’t join the rota. As I mentioned in <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">my previous post</a>, part of the solution to this involves doing the hard work to make the tech simpler, such as swapping out the complex parts of the codebase.</p>
<p>However, we also did a lot of great work to make it easier to be on the out of hours rota:</p>
<ul>
<li>We had a <a href="https://medium.com/ft-product-technology/documentation-day-how-the-ft-com-team-improved-our-documentation-to-95-usefulness-in-7-hours-b73d1a7e6f30">documentation day</a> to make sure the documentation people look at for support with systems in an incident were up to date.</li>
<li>We ran <a href="https://medium.com/ft-product-technology/supporting-our-systems-through-incident-workshops-c0d22a4290b9">incident workshops</a> so people new to incidents could understand what was involved. Before you’ve been involved in support incidents sometimes you feel that you need to know everything about the stack you are supporting in order to sign up, and one of the really great benefits of the incident workshops is that less experienced staff saw more senior staff saying they didn’t know what was happening, and then see what things they’d investigate to figure it out.</li>
<li>We introduced a shadow rota, where people could sign up to the out of hours rota but as a shadow, meaning they would get called when there was an incident, but they would not be on the hook for solving it; they would observe and learn. We found that people on the shadow rota often make really valuable contributions to handling incidents.</li>
<li>We’ve also just introduced a slack group called <code class="language-plaintext highlighter-rouge">@incident-watchers</code> so that people who want to learn about live incidents can be tagged when we’re having one in-hours and watch along.</li>
</ul>
<p>These changes meant that we went from 5 to 22 people on the out of hours rota, which is much more sustainable.</p>
<p><img src="/img/incident_workshop.png" alt="A group of people sat round a table with laptops and metrics displayed" /></p>
<h2 id="achieving-operational-excellence">Achieving operational excellence</h2>
<p>Improving the out of hours support process is great, but what is even better is improving the reliability of the site so that there are fewer failures and it is quicker to recover when there are.</p>
<p>When I joined, we had a rotating responsibility for improving reliability called Ops Cops, which teams would take on for a week at a time, We’ve made a few incremental changes to how we do this, each improving our reliability.</p>
<p>Firstly, we <a href="https://medium.com/ft-product-technology/why-and-how-we-changed-the-way-we-support-ft-com-7572371f3f36">created a dedicated team with a tech lead and delivery manager, that engineers rotated into</a> so there was consistency from week to week.</p>
<p>While this improved things, we found that the rotation meant that the team was not able to focus on larger, more strategic reliability work. The team also didn’t have clear metrics or service level agreements, so we then <a href="https://medium.com/ft-product-technology/next-chapter-on-our-journey-to-achieve-and-maintain-operational-excellence-of-ft-com-7dd9c7871347">created a stable team of three engineers</a> and a delivery manager, and we added a product manager.</p>
<p>One year on from these changes <a href="https://medium.com/ft-product-technology/operational-excellence-one-year-on-f1dccaf2f034">the team are doing great things</a>; improving observability and reliability, working more proactively rather than reactively, and making recommendations and providing tools that help us achieve operational excellence.</p>
<h2 id="improving-technical-alignment">Improving technical alignment</h2>
<p>We have multiple teams working on microservices who are empowered to make technical decisions. However, teams were getting out of alignment with each other. Different teams were making technical decisions that ultimately lead to conflicting or unclear architectural changes.</p>
<p>To address this, we introduced a <a href="https://medium.com/ft-product-technology/thinking-through-our-changes-with-design-documents-35c93bf0938b">technical design document process</a>. When tech leads are making architectural decisions, they share these documents with the other tech leads for discussion. This allows people with relevant experience in a different team to add useful information to a plan, whereas otherwise they might not know it is happening.</p>
<p>As a side benefit it also forms a documentation trail of architectural decisions. We also now use these when other teams would like to make changes to our stack.</p>
<h2 id="we-are-part-of-a-larger-team">We are part of a larger team</h2>
<p>In 2022, we’ve also been able to expand our tech strategy to be much more closely supporting the goals of the business. When we <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">first started working on the Customer Products tech strategy</a> in 2018, the vision for FT.com was still around the recent launch of the new site, and our tech strategy was focused on making sure that it remained sustainable. It was about building good tech and not costing the business more money, but it wasn’t about delivering specific products that move us forward.</p>
<p>The best tech strategy could be summarised as “deliver what the business needs”, and now, we have a <a href="https://medium.com/ft-product-technology/financial-times-subs-and-product-strategy-2021-1bc48a2a337e">subscriptions business strategy</a>, and a clear and compelling <a href="https://medium.com/ft-product-technology/filling-the-product-strategy-gap-22fb6f792139">product strategy for FT.com & apps</a> which helps us really focus our tech strategy on moving towards the most important goals for the business.</p>
<p>In addition, we are now working much more closely with other tech groups across the FT to pull together a tech strategy for all of FT Technology, rather than each group having their own, separate tech strategy.</p>
<h2 id="whats-next-for-no-next-next">What’s next for no next Next</h2>
<p>We’ve seen that our tech strategy has really bedded in and allowed people to make decisions based on what helps make our codebase more sustainable, and enabled and supported the great ideas the team have.</p>
<p>Over time, we’ve changed how we work on tech strategy, from setting <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">quarterly priorities</a>, to this year, following a <a href="/jfdi/tech-strategy-process.html">deeper, more forward-looking process that helped us set priorities for the whole of 2022</a>. Our process in 2022 allowed us to really invest more in our tech strategy priorities. For example, building a team to work on API rationalisation for six months, rather than trying to do this work alongside other, competing priorities, and we’ve really seen the benefit of that approach over this year.</p>
<p>I mentioned that Next brought all of the website estate under one leadership team, meaning that product, UX and design work together to form a coherent user experience, and the tech is governed in one place.</p>
<p>As we move towards a multi-product organisation the exciting challenge for Customer Products is how to enable other tech groups to deliver features, some of which will be on FT.com and the apps, without losing the coherent user experience and the strong tech governance.</p>
<p>All this means that the 2023 Customer Products tech strategy will be part of a larger business strategy. So there may be no next for No next Next, but I’m incredibly excited about what we are going to do next!</p>
<p><em>This post originally appeared on the <a href="https://medium.com/ft-product-technology/what-next-for-no-next-next-adbb02406171">FT Product & Technology blog</a></em></p>
Working together on tech strategy2022-12-09T00:00:00+00:00https://www.annashipman.co.uk/jfdi/tech-strategy-process.html<p>When working on tech strategy I like to involve as wide a group as possible. During the pandemic, with remote and hybrid working, this become more difficult. This post is about a way we found to define the 2022 tech strategy that worked well.</p>
<h2 id="its-important-to-involve-a-wide-group-in-tech-strategy">It’s important to involve a wide group in tech strategy</h2>
<p>The ideal when working on tech strategy is to involve a wider group of senior and principal engineers as well as representatives from other disciplines, e.g. product and design. There are two main reasons for this.</p>
<p>The first is that input from the people doing the work leads to better decisions.</p>
<p>The second is that a big part of creating and implementing a strategy successfully is having a shared understanding of what you are are doing and why, and being involved in that process means we all understand that better.</p>
<p>I can share context about the business and what’s coming up organisationally, and the wider group know about the detail of the day to day work, technical challenges and opportunities.</p>
<p>The more people understand what we are trying to do and why and are involved in the creation of strategy, the better able people are to make independent decisions about the right thing to do.</p>
<h2 id="previously-we-clarified-the-bigger-picture-through-a-card-laying-exercise">Previously, we clarified the bigger picture through a card-laying exercise</h2>
<p>In my <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">previous post about setting a tech strategy</a> I described the process we used on Customer Products at the FT (the team in charge of FT.com and the apps) to work out what highest leverage actions we should take to move us towards our vision of sustainable, supportable, simple tech.</p>
<p>One of the activities we did was a card-laying exercise, where we laid cards out on the floor and rearranged them to work out what the priorities were. That exercise gave us clear outline and a shared understanding of where we were heading for a few years.</p>
<p>Over the next two years, we had quarterly, shorter meetings, where we refreshed where we had got to and looked forward to what the next steps might be.</p>
<p>However, after a couple of years, we were getting to the point where we needed to have another card-laying exercise, with everyone involved, and we booked an Away Day for March 24th, 2020.</p>
<p>However, on March 23rd, 2020, the UK went into lockdown. The Away Day was cancelled, and we did not arrange another one. Partly because I could not work out how to do a card-laying exercise remotely, and partly because a full day remote meeting is a horrible thing to ask people to do.</p>
<h2 id="during-the-pandemic-we-tried-various-different-ways-of-working-on-the-strategy">During the pandemic we tried various different ways of working on the strategy</h2>
<p>We couldn’t have the full day meeting we had planned, and in the first, very stressful, few months of the pandemic, it didn’t feel possible to think about how to plan ahead.</p>
<p>Once we were able to do so, we continued with our quarterly meetings, and tried different ways to prioritise remotely.</p>
<h3 id="discussion-and-voting">Discussion and voting</h3>
<p>The quarterly meetings in the office had involved a discussion, proposing priorities on cards and then voting. The discussions, however, usually were two or more hours, and it is much harder to have a long and constructive meeting remotely.</p>
<p>We attempted to replicate this by populating a Trello board with cards ahead of time, then discussing in the meeting and voting using a Trello plug-in.</p>
<p>This had the advantage of being a shorter meeting, but we found that it was quite easy to sway the meeting; often the priorities ended up being something that one or two people could speak passionately about, rather than the things that actually had the highest leverage.</p>
<p>This was something that had also happened in the in-person meetings as well, but because there was more time for discussion when we had the meetings in-person, this effect was slightly mitigated.</p>
<p>In addition, it was harder to get everyone to participate in these remote meetings than it had been in person; body language, moving around the room, the ability to have quiet side conversations were all missing.</p>
<h3 id="deciding-within-a-smaller-group">Deciding within a smaller group</h3>
<p>Because we still felt we were unable to prioritise bigger, more impactful pieces of work we also had some smaller discussions with just myself and the principal engineers to identify priorities. <a href="/jfdi/prioritising-in-tech-leadership.html">I’ve written that up here</a>.</p>
<p>The advantage of this was we were able to make some big decisions because it’s easier to discuss in a smaller group, but the big disadvantage was that we didn’t involve the wider group, so we missed out on their input and they missed out on the context.</p>
<h3 id="focusing-on-specific-activities">Focusing on specific activities</h3>
<p><a href="https://alicebartlett.co.uk/">Alice Bartlett</a> led two remote tech strategy meetings with the wider group focusing on specific activities. In one, smaller groups worked on assessing our capabilities and identifying our pain points, and in the other the team worked on drawing our software architecture using the <a href="https://c4model.com/">C4 model</a>.</p>
<p>These were really useful for context-sharing, surfacing issues and getting agreement, but they were less useful for setting the clear priorities and next steps, or clarifying the overarching most important themes.</p>
<p>We were able to move forward with the strategy, and do some great work, but we still felt we were missing out on the bigger picture, shared with the larger group.</p>
<h2 id="changing-the-process-for-2022">Changing the process for 2022</h2>
<p>At the beginning of 2022 we decided to try a new process. I had initially been looking to get a professionally facilitated remote workshop and have a full day event, but as I started preparing what we would need to share ahead of that workshop, it occurred to me that we could learn from what had worked and what hadn’t.</p>
<p>One thing we know is that meetings are not the best way to get input from everyone. Some people don’t feel comfortable speaking up in group meetings, and some people need time to think things through and don’t have their best ideas in the meeting, but hours, or days later while reflecting.</p>
<p>So for 2022 we did it completely differently.</p>
<h2 id="first-we-had-a-meeting-purely-to-share-the-context">First, we had a meeting purely to share the context</h2>
<p>This time, our quarterly tech strategy meeting was explicitly and purely for context sharing. We invited guest speakers and the purpose of the meeting was to share context of the various strategies that affect our work, discuss and ask questions.</p>
<p>For example, <a href="https://www.linkedin.com/in/debbie-mcmahon-865529116/">Debbie McMahon</a>, Product Director for FT.com and apps, talked about the <a href="https://medium.com/ft-product-technology/financial-times-subs-and-product-strategy-2021-1bc48a2a337e">subscriptions strategy</a> and the FT.com & apps product strategy. We also talked about the wider tech strategy, across all groups, and any relevant OKRs. We then talked about our plan for working on the Customer Products tech strategy in 2022</p>
<h2 id="we-then-shared-a-document-with-possible-areas-of-focus">We then shared a document with possible areas of focus</h2>
<p>I and the Principal Engineers put together a document outlining our prioritisation framework, and then a list of all the areas of focus that we thought might be worth prioritising. The document was 8 pages long with about 12 possible areas we could focus on and some bullet point details about why each was important and the impact it could have.</p>
<p>We shared the document with the wider group: senior engineers, tech leads, principal engineers, product managers, delivery managers, designers and user researchers.</p>
<p>We asked for comments, corrections, and to add additional things we should cover, and their thoughts on priority.</p>
<p><img src="/img/areas_of_focus.png" alt="Screenshot of the beginning of the document, noting that it is a proposal, that we want input, and asking for comments and feedback" /></p>
<p>Interestingly, one early comment we got was on our prioritisation framework. Initially this was:</p>
<ol>
<li>Supports FT.com & apps product strategy for 2022</li>
<li>Something we should invest in now to put us in a good place in 3-5 years (This is <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">No next Next</a>, our Customer Products tech strategy)</li>
<li>Something that supports and/or shapes the overall FT tech strategy</li>
</ol>
<p>But <a href="https://twitter.com/binaryberry">Tatiana Stantonian</a> made the excellent observation that this ordering very strongly implies that the top priority has to be feature delivery for 2022. But part of the purpose of even having a tech strategy is to make sure that we are prioritising the investments we need to make to be in a good situation in the future.</p>
<p>So we swapped the ordering of 2 and 1.</p>
<h2 id="we-then-had-six-smaller-meetings">We then had six smaller meetings</h2>
<p>After people had some time to digest and comment on the document, each Principal Engineer (of whom there were six at the time) set up a meeting with the tech leads, senior engineers, product managers and other interested parties for each of the teams they oversee.</p>
<p>In that meeting, we talked through the document and got feedback.</p>
<p>The advantages of the smaller meeting was it was easier for people to contribute, and everyone did; there wasn’t anyone in any of the meetings who didn’t say anything. And because people had had time to digest the document, they were able to give good input in the discussion.</p>
<p>There were two main disadvantages to this approach. Firstly, because the process was a document, then six meetings, it took longer overall to get to done than an Away Day would have; however, the content was better because people do not always have their best ideas on the day.</p>
<p>The other disadvantage was that because we split it into six discussions, the only person who had the context of the whole lot was me, as I was in every conversation. We mitigated that by the principal engineer making notes on each meeting which we shared with everyone, so people could at least see what was discussed in the other groups.</p>
<p>Finally, after we’d broken out into the six meetings, we pulled it all back together, outlined what our top two priorities were, and shared it at an all staff meeting as well as in the document.</p>
<h2 id="the-feedback-on-the-new-process-was-positive">The feedback on the new process was positive</h2>
<p>There was generally positive feedback on this approach. 50% of those who attended returned the feedback form and all of them answered yes to the questions “did you find the meeting useful?”, “did you find the meeting interesting?”, “did you feel you were heard in this meeting?”. To the question “Do you feel you have a good idea of the Customer Products tech strategy?” 80% answered yes, with the other 20% answering “partly”.</p>
<p>There was also a lot of useful comment and suggestions for improvement, mainly around making it more focused in the conversation (perhaps with homework beforehand) but also how to make sure we shared the context across the groups – the notes were useful, but some people wished they’d been able to join multiple discussions. One point made was that the document possibly made it feel too concrete; less of a discussion, more of a plan.</p>
<p>There were also useful comments on what people would like from the Customer Products tech strategy, mainly around clarity on what it meant for each person and team. This comment was a representative and insightful one: it should offer “clarity on what we should prioritise (and therefore what to push back on). A feeling of moving forward and not only clearing up previous mess.”</p>
<h2 id="the-results-over-the-year-were-also-positive">The results over the year were also positive!</h2>
<p>The big advantage of doing it this way meant that we had clear priorities for the whole year, and that meant we were able to invest in the areas we’d prioritised. For example, we identified as our top priority rationalising our content APIs and we were able to <a href="https://medium.com/ft-product-technology/unspaghettiing-ft-coms-content-pipeline-be1421a434cb">build a team and give them six months to work on it</a>, i.e. properly invest in things that put us in a good place for future.</p>
<p>We identified two top priorities, and we also identified that one of the things we had deprioritised was very important but we did not have the right capacity to work on it. This meant that when an opportunity emerged to focus on that, it was an easy decision to change the order, deprioritise the second thing we’d agreed to work on and and focus instead on the top ‘unprioritised’ piece of work.</p>
<p>There were also advantages to having listed out and discussed all the areas of focus. With that shared context, when opportunities came up to move forward on some of these, teams knew that it was valuable. So even pieces of work that had not been deemed top priority were able to progress. For example, the reliability team did a huge amount of work on <a href="https://medium.com/ft-product-technology/operational-excellence-one-year-on-f1dccaf2f034">operational excellence</a>.</p>
<p>This process meant it was much easier to get product involvement, which is really important for a successful and useful tech strategy. We were also able to share this outside of the Customer Products team which helped others understand our tech strategy in more detail than usual. Not everyone likes to read long documents, but it was useful for those who do.</p>
<p>One thing that it missed out was creating the forum for ideas generation. People did add suggestions to the document of things we might consider, but they were usually ones they’d already been thinking of. One advantage of a big group discussion is, if done right, it can really create the environment to spark new ideas and thinking. It would be good to incorporate something there in future.</p>
<h2 id="we-will-probably-do-a-similar-process-next-year">We will probably do a similar process next year</h2>
<p>On balance, this worked very well, so we will definitely use parts of this process next year, though next year we will have more inputs to draw in, and we are very close to having a tech strategy for the whole of the FT not just the separate groups. I’m really excited to see what we do next!</p>
Seven Concurrency Models in Seven Weeks2022-07-16T00:00:00+00:00https://www.annashipman.co.uk/jfdi/7-concurrency-models-7-weeks.html<p>Last year, I read the excellent and very interesting <a href="https://pragprog.com/titles/pb7con/seven-concurrency-models-in-seven-weeks/">Seven Concurrency Models in Seven Weeks</a> (though in somewhat more than seven weeks). This is a very long post with my notes from reading it.</p>
<h2 id="tldr-you-can-follow-along-with-the-book-using-my-vagrantfile">TL;DR: you can follow along with the book using my Vagrantfile</h2>
<p>I created a <a href="https://github.com/annashipman/7weeks-concurrency">Vagrantfile</a> to run the code samples in. You can use this to follow along with the book.</p>
<p>There are a couple of places where the code has to be changed in order to run but I have left that as an exercise to the reader; the error messages tell you enough.</p>
<p>There are some examples (e.g. Hadoop) that won’t work on the Vagrant box.</p>
<h2 id="there-is-a-difference-between-concurrent-and-parallel">There is a difference between concurrent and parallel</h2>
<p>Concurrent and parallel are different but related concepts. A concurrent program has multiple logical threads of control. A parallel program is executing different parts of the computation simultaneously, and may or may not have more than one logical thread of control.</p>
<div class="code-sample">
<p>Let's say we are tidying the house.</p>
<p>Concurrency: I do the washing up while you hoover the house.</p>
<p>Parallelism: I hoover downstairs and you hoover upstairs at the same time.</p>
</div>
<p>Another way of thinking about it is that concurrency is an aspect of the problem domain: your program needs to handle multiple simultaneous events. Parallelism is an aspect of the solution domain; you want to make your program faster by processing different parts of the problem in parallel.</p>
<h2 id="week-one-threads-and-locks">Week One: Threads and Locks</h2>
<p>The “first week” is about threads-and-locks programming. This is a really great intro to concurrency because it is very low-level; “little more than a formalization of what the underlying hardware actually does”, so it helps you to get a foundational understanding of what’s going on.</p>
<p>I’ve included more notes on this as some of the key themes of concurrent programming are included in this section.</p>
<p>The book points the reader towards the <a href="http://www.cs.umd.edu/~pugh/java/memoryModel/jsr-133-faq.html">FAQ for JSR 133</a>. A JSR is a Java Specification Request, and JSR 133 updates the original Java Memory Model. It explains what a memory model does at the processor level, and explains important ideas, like for example the fact that a compiler might move a write operation earlier or later than it appears in the written program, to improve performance.</p>
<p>Java includes language constructs like <code class="language-plaintext highlighter-rouge">volatile</code> and <code class="language-plaintext highlighter-rouge">synchronized</code>, which are intended to help the programmer describe a program’s concurrency requirements to the compiler. However, you have to make sure you are using them correctly and it’s very easy to incorrectly synchronise your program so it is not threadsafe.</p>
<h3 id="intrinsic-locks">Intrinsic locks</h3>
<p>Every Java object has an intrinsic lock. However, there are problems with the intrinsic locks. For example, you can’t interrupt a thread that is blocked trying to acquire a lock – the only way to do this is to kill the JVM. So Java introduced other locking techniques, for example <code class="language-plaintext highlighter-rouge">ReentrantLock</code> which allows interruptible locks and timeouts. Though this is still not a perfect solution; for example it can lead to <code class="language-plaintext highlighter-rouge">livelock</code> where all the threads time out at the same time and become deadlocked again.</p>
<p>The book then talks through some other locking mechanisms, for example hand-over-hand locking (where you lock a small portion, e.g. of a linked list), fairness parameter on <code class="language-plaintext highlighter-rouge">ReentrantLock</code> (the lock obeys the order of the lock request) and <code class="language-plaintext highlighter-rouge">Condition</code> variables (allow you to wait on the thing you need. The thread acquires the lock and checks to see if the thing is ready. If it’s not, it releases the lock and waits again).</p>
<h3 id="atomic-variables-are-an-alternative-to-locks">Atomic variables are an alternative to locks</h3>
<p>Another option instead of intrinsic locks is atomic variables, for example <code class="language-plaintext highlighter-rouge">AtomicInteger</code> counter will read a value and increment atomically. This is better than using a lock because you can’t forget to acquire the lock; and also it’s impossible for it to deadlock.</p>
<p>Atomic variables are the foundation of non-blocking, lock-free code.</p>
<h3 id="dont-roll-your-own-locking-solution">Don’t roll your own locking solution</h3>
<p>While it’s important to understand how locks work, it’s better to use concurrent data structures rather than roll your own locking solution. For example, if you create a thread for each incoming connection, you might run out of processors, and it’s a perfect attack vector for a DoS. Better is to create a thread pool of a certain size (e.g. approximately same number of threads as processors for computation intensive tasks, or perhaps twice as many threads as processers for IO-intensive tasks). If more requests are active, they will be queued until a thread becomes free; so we won’t grind to a halt – and we also won’t incur the cost of starting a new thread for each request.</p>
<h3 id="an-example">An example</h3>
<p>Throughout the book, we return to the same example of how we might speed something up by parallelising. The example is counting words in an XML dump of Wikipedia, using the producer-consumer pattern. The producer produces the pages from the XML dump. The consumer consumes those pages and counts the words.</p>
<p>The producer is very quick. So to speed things up, we could parallelise the consumer. However, if we are going to do that, we need to find a way to synchronize access to the counts map.</p>
<p>We could lock within each count. But what that does is mean that each consumer spends more time waiting for the map to be unlocked than they do actually do doing work (excessive contention).</p>
<p>We can use <code class="language-plaintext highlighter-rouge">ConcurrentHashMap</code>, which provides atomic read-modify-write methods, and has also been designed to support high levels of concurrent access via a technique called lock-striping. We can speed it up even further by batching the jobs, i.e. each consumer keeps a local word count and we merge them when we exit the program.</p>
<p>Instead of a thread pool you might use a <code class="language-plaintext highlighter-rouge">ForkJoinPool</code>, which works by ‘work-stealing’; this means the threads actively look for tasks to execute in the pool. This makes for efficient processing when the majority of the other tasks spawn small sub-tasks, or many small tasks are submitted.</p>
<p>A <code class="language-plaintext highlighter-rouge">CyclicBarrier</code> is something that allows a set of threads to all wait for each other until they’ve all reached a common barrier point (‘cyclic’ means it can be reused). A <code class="language-plaintext highlighter-rouge">CountDownLatch</code> is another way of performing a similar action: in this case you set a count and once that number of threads has called the countDown() method it releases all waiting threads.</p>
<h3 id="there-is-a-limit-to-how-much-speedup-parallelising-can-give-you">There is a limit to how much speedup parallelising can give you</h3>
<p>If you time the speedup you get from parallelising, what you see is: performance initially increases linearly. It is then followed by a period where it increases more slowly. Eventually it will peak and adding more threads will only make it slower.</p>
<p><a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl’s Law</a> is a formula that gives the theoretical speedup that parallelising a program can give. It relates to how much of the program can be parallelised.</p>
<p>Long after I’d finished reading the book but before I’d finished this write-up, I attended a brilliant keynote at QCon on <a href="https://qconlondon.com/london2022/keynote/future-microprocessors">the future of microprocessors</a> by <a href="https://royalsociety.org/people/sophie-wilson-12544/">Sophie Wilson</a>. In it she talked about Amdahl’s law and noted that what most people miss about it is how aggressive it is.</p>
<p><img src="/img/AmdahlsLaw.svg" alt="Graph showing theoretical speedup of a program as a function of number of processors" />
By <a href="https://en.wikipedia.org/wiki/User:Daniels220" class="extiw" title="wikipedia:User:Daniels220">Daniels220</a> at <a href="https://en.wikipedia.org/wiki/" class="extiw" title="wikipedia:">English Wikipedia</a>, <a href="https://creativecommons.org/licenses/by-sa/3.0" title="Creative Commons Attribution-Share Alike 3.0">CC BY-SA 3.0</a>, <a href="https://commons.wikimedia.org/w/index.php?curid=6678551">Link</a></p>
<p>The graph shows that where 95% of the program is parallelisable, you can get it up to 20 times faster. Where 90% is parallelisable, this drops to only 10 times faster. And if only 50% of the program is parallelisable, the most speedup you can get is only 2x.</p>
<h3 id="strengths-and-weaknesses-of-threads-and-locks-programming">Strengths and weaknesses of threads-and-locks programming</h3>
<p>The main strength of threads-and-locks programming is that it’s little more than the hardware offers so can be very efficient when applied correctly.</p>
<p>However, you don’t get much support on top of that; and it only supports shared-memory architectures, so can’t be used to solve distributed problems. Threads-and-locks programming also doesn’t provide direct support for parallelism, only concurrency.</p>
<p>But what makes it really hard is that it is impossible to test. Bugs tend to be intermittent and infrequent and may take a long time to show up, and there is no way to write automated tests that validate you’ve executed threading correctly. Because of that latter point, you can’t reliably refactor the code.</p>
<h2 id="week-two-functional-programming">Week Two: Functional programming</h2>
<p>Functional programming avoids many of the problems with threads-and-locks by avoiding mutable state.</p>
<p>An imperative program consists of a series of statements that change global state when executed. In contrast, a functional program models computation as the evaluation of expressions. These expressions are both first-class (i.e. can be manipulated like any other value) and do not have side effects. This lack of side effects makes reasoning about thread safety much easier. Immutable data dosn’t need to be locked because multiple threads can’t change it.</p>
<p>You can’t always tell when you have mutable state. It might be hidden, e.g. in a function you are calling. He also gives an example of escaped mutable state, where it looks at first glance that everything is synchronised that should be, but one method returns an iterator, and the iterator still has access to the mutable state it is iterating on.</p>
<h3 id="side-note-my-vagrantfile-wasnt-working-exactly-as-planned">Side note: My Vagrantfile wasn’t working exactly as planned</h3>
<p>While I could run the Java examples in my Vagrant box, the Clojure examples did not work as expected. I <a href="/jfdi/clojure-vagrant.html">wrote all that up in a separate post</a>; the summary is, I could not get these examples working correctly but learned a lot about Clojure in the process.</p>
<p>For example, sequences are lazy; as well as not evaluating later parts of a sequence until and unless we need them, it also discards the elements at the front if we’ve finished with them (we don’t “hold on to our head”). This means if you run something like <code class="language-plaintext highlighter-rouge">take-last 5 (range 0 100000000)</code> might take a while to run but won’t cause you to run out of memory.</p>
<p>This <a href="https://clojure.org/api/cheatsheet">Clojure cheatsheet</a> is great.</p>
<h3 id="functional-programming-allows-us-to-parallelise-code-more-easily-because-it-is-declarative">Functional programming allows us to parallelise code more easily because it is declarative</h3>
<p>In imperative languages like Java, things happen roughly in the order you write them, leaving aside some moving around by the compiler and runtime.</p>
<p>In functional languages, it is declarative. Instead of writing a set of instructions for how to perform some operation, you give a statement of what the results should be, so there is more freedom to reorder calculations.</p>
<p>Pure functions are referentially transparent; this means that anywhere an invocation of the function appears, we can replace it with its result without changing the behaviour of the program. in fact one way to think about what executing functional code means is to think of it as repeatedly replacing function invocations with their results until you reach the final result. Because <em>every</em> function is referentially transparent, we can safely make changes to evaluation order, and thus we can easily parallelise.</p>
<p>The primary benefit of functional programming is confidence that your program does what you think it does. It can be difficult to get into the mindset but once you have, functional programs tend to be simpler, easier to reason about and easier to test than their imperative equivalents.</p>
<h3 id="futures-and-promises">Futures and promises</h3>
<p>Dataflow programming is the idea that you push data towards functions that need them for their input. So you can execute the functions that do not have interdependencies concurrently, but some functions will need to wait for those to be executed before they can start.</p>
<p>Clojure allows us to use this execution strategy through futures and promises.</p>
<p>A future takes a body of code and executes it in another thread. The return value is a future object, and we get the value by dereferencing it using <code class="language-plaintext highlighter-rouge">deref</code> or <code class="language-plaintext highlighter-rouge">@</code>. This will block until the value is available.</p>
<p>A promise is very similar to a future, but creating a promise does not cause any code to be run. Instead its value is set with <code class="language-plaintext highlighter-rouge">deliver</code>.</p>
<p>A classic case for futures is talking to another web service. A future allows computation, such as network access, to take place on another thread while the main thread continues.</p>
<p>I loved the <a href="https://ericnormand.me/guide/clojure-concurrency">summary of concurrency in Clojure at the start of this tutorial</a>.</p>
<h2 id="week-three-the-clojure-way-aka-what-about-mutable-state">Week Three: The Clojure Way (a.k.a. what about mutable state?!)</h2>
<p>Some problems have modifying state as an inherent part of the solution. For this there is the “Clojure way”; the idea is that it uses functional programming <em>and</em> mutable state to get the best of both worlds.</p>
<p>A “pure” functional language provides no support for mutable data at all, so Clojure is “impure”. It provides a number of different concurrency-aware mutable variables.</p>
<p>The difference between an impure functional language and an imperative language is emphasis. In an imperative language, variables are mutable by default and idiomatic code modifies them frequently. In an impure functional language, variables are immutable by default and idiomatic code modifies those that aren’t only when absolutely necessary.</p>
<h3 id="data-structures-are-persistent">Data structures are persistent</h3>
<p>All of Clojure’s data structures are persistent. This means that they always preserve the previous version when they are modified, so code can have a consistent view of the data in the face of modifications.</p>
<p>Persistent data structures separate identity from state. For example, a thread might have a reference to a data structure – the identity. The state may vary. This is different from an imperative language when a single identity has a single value, making it easy to lose sight of the fact that state is really a sequence of values over time. He quotes Heraclitus “You could not step twice into the same river; for other waters are ever flowing onto you”.</p>
<h3 id="supporting-shared-mutable-state">Supporting shared mutable state</h3>
<p>To support shared mutable state in Clojure, you can use atoms, agents, or refs.</p>
<ul>
<li>Atoms. An atom is an atomic variable, very similar to Java’s atomic variables – in fact Clojure atoms are built on <code class="language-plaintext highlighter-rouge">java.util.concurrent.atomic</code> An atom allows you to make independent, synchronous changes to a single value – synchronous because when <code class="language-plaintext highlighter-rouge">swap!</code> returns, the update has taken place</li>
<li>Agents. Like atoms in that they encapsulate a reference to a single value which can be deferenced and is changed using <code class="language-plaintext highlighter-rouge">send</code>. However, <code class="language-plaintext highlighter-rouge">send</code> returns immediately, before the value of the agent has been changed, and the function passed to <code class="language-plaintext highlighter-rouge">send</code> is called some time later. Asynchronous updates have benefits, especially for long-running operations. But they also have added complexity.</li>
<li>Refs are more sophisticated – providing <a href="https://en.wikipedia.org/wiki/Software_transactional_memory">software transactional memory</a> (STM), which means we can make changes to to multiple variables; concurrently and in a co-ordinated fashion. However, they have to be changed as part of a transaction. Transactions are ACI (atomic, consistent and isolated) but not D (durable). As per database transactions. If you need durable; use a database instead.</li>
</ul>
<p>How to choose between these approaches? To a certain extent it’s a matter of personal preference, but even though STM gets the headlines, atoms suffice for most problems, as the language’s functional nature leads to minimal use of mutable data. “As always, the simplest approach that will work is your friend”.</p>
<h2 id="week-four-actors">Week Four: Actors</h2>
<p>Actors do away with shared mutable state altogether. Functional programming avoids the problems associated with shared mutable state by avoiding mutable state. Actor programming retains mutable state, but avoids sharing it.</p>
<p>An actor is a bit like an object in Object-Oriented programming. It encapsulates state and communicates with other actors by exchanging messages. The difference is actors run concurrently and they really do send messages (as opposed to OO where it’s actually just calling a method).</p>
<p>For this chapter, the language used is Elixir (which runs on the Erlang JVM. Fun fact! Erlang uses <a href="https://github.com/elixir-lang/elixir/issues/9269">British spellings</a>, good reason to use it!).</p>
<p>In Elixir, an actor is called a process. In most environments, a process is a heavyweight entity that consumes a lot of resources and is expensive to create. In Elixir however, it’s lightweight, lighter weight than most system threads, and Elixir programs typically create thousands of processes without problems. An actor is like a rental car. It’s easy to get hold of, and you don’t fix it if it breaks down, you just get another one.</p>
<p>You define an actor, the messages it knows how to receive and what it does when it receives them.</p>
<p>One of the most important features of actor programming is that messages are sent asynchronously, and placed in a mailbox. An actor handles messages sequentially, in the order they were added to the mailbox, moving onto the next message only when it’s finished processing the current message.</p>
<p>A supervisor is a system process that monitors one or more worker processes and takes appropriate action if they fail. This can include restarting it. Actor programs tend to avoid defensive programming and instead let it crash, allowing the actor’s supervisor to address the problem instead. This has multiple benefits, including:</p>
<ul>
<li>code is simpler and easier to understand, with a clear separation between ‘happy path’ and fault-tolerant code</li>
<li>actors are separate from one another and don’t share state, so there’s little danger that a failure in one actor will adversely affect another</li>
<li>As well as fixing the error, the supervisor can log it so we become aware of problems</li>
</ul>
<p>One of the actor model’s primary benefits is that it supports distribution. Sending a message to an actor on another machine is just as easy as sending it to one running locally.</p>
<p>There is a library called OTP which automates the creation of the underlying workers. (OTP is Open Telecom Platform as Erlang started out in telecommunications, but very little of it is now telecom-specific so it’s now just “OTP”).</p>
<p>In a way, actors are more object-oriented than OO objects, with stricter method-passing and encapsulation. Actors do not share state, and although they run concurrently with each other, within a single actor everything is sequential, which means we only need to worry about concurrency when considering message flows between actors. So actors can be tested in isolation, and if we find a concurrency bug we know it’s in message flows.</p>
<p>Distributed programming means an actor program can scale to solve problems of almost any size, we are not limited to problems that fit on a single system. However, there is not direct support for parallelism. Parallel solutions need to be built from concurrent building blocks, raising the spectre of non-determinism.</p>
<h2 id="week-five-communicating-sequential-processes">Week Five: Communicating Sequential Processes</h2>
<p>Like functional programming and actors, CSP is an old idea that has experienced a renaissance, initially because of Go.</p>
<p>CSP is about focusing on the roads, rather than the cars. The features and capabilities of a message-passing systems are not primarily defined by the code between which messages are exchanged, or their content, but by the transport over which they travel.</p>
<p>In CSP channels are first class. Instead of each process being tightly coupled to a single mailbox, channels can be independently created, written to, read from and passed between processes.</p>
<p>A channel is a threadsafe queue. Any task with a reference to a channel can add messages to one end, and any task with a reference to it move messages from the other. Unlike actors, where messages are sent to and from specific actors, senders don’t have to know about receivers or vice versa.</p>
<p>Treating the channels as first class gets rid of callback hell in two areas where there is traditionally a lot: asynchronous IO and UI programming.</p>
<p>One weakness of CSP is that there is not as much focus on distribution and fault tolerance. Also, as with both threads-and-locks and actors, CSP programs are susceptible to deadlock and have no direct support for parallelism.</p>
<p>Most of the difference between actors and CSP result from the differing focus of the communities: Actors has focused on fault tolerance and distribution and CSP on efficiency and expressiveness, so choosing between them is largely a question of deciding which of these aspects is most important to you.</p>
<h2 id="week-sixdata-parallelism-using-the-gpu">Week Six: Data parallelism using the GPU</h2>
<p>CSP was the last of the general-purpose programming models, and for the last two chapters, we looked specifically at data parallelisation.</p>
<p>The first one is using the Graphics Processing Unit (GPU). The GPU is a powerful data-parallel processor, and can be much faster than the CPU when used specifically for number-crunching. This is called general purpose computing on the GPU, or GPGPU (!!)</p>
<p>Computer graphics is all about manipulating data very quickly. A scene in a 3D game is made up of loads of tiny triangles that need to have their position calculated, lit, textured, etc, many times a second, and modern GPUs are capable of manipulating billions of these per second. Although the amount of data that needs to be processed is huge, the relative operations on that data are relatively simple vector or matrix operations, so they are amenable to data parallelisation, where multiple computing units perform the same operations on different items of data in parallel.</p>
<p>Data parallelisation can be implemented in many different ways. The book looks at two: pipelining and multiple ALUs.</p>
<ul>
<li>Pipelining. At the level of gates on a chip, something as simple as multiplying 2 numbers takes several steps. It may take say, 5 clock cycles. However, if we have multiple operations, we can just pack them in behind each other and keep it moving. So multiplying a thousand pairs of numbers takes a bit over a thousand clock cycles rather than five thousand clock cycles.</li>
<li>Multiple ALUs. An ALU is an arithmetic logic unit. If you have a lot of these and a wide memory bus that allows multiple operands to be fetched simultaneously, operations on large amounts of data can be parallelised.</li>
</ul>
<h3 id="each-gpu-is-architected-slightly-differently">Each GPU is architected slightly differently</h3>
<p>GPUs use a combination of these approaches amd many others to achieve performance, and each GPU is architected slightly differently.</p>
<p>Open Computing Language (OpenCL) abstracts away the details of the GPU implementation by defining a C-like language which allows us to express a parallel algorithm abstractly. Each different GPU manufacturer then provides its own compilers and drivers that allow the program to be compiled and run on its hardware.</p>
<p>To parallelise our array multiplication task we need to divide it into work-items that will be executed in parallel. These are typically very small, as small as they possibly can be (unlike other parallelisation, where you might want to ensure each task is not <em>too</em> small so it doesn’t waste effort creating threads and communicating). The OpenCL compiler and runtime then worry about how best to schedule these work-items.</p>
<p>We specify how each work-item should be processed by writing an OpenCL kernel. You then need to embed the kernel in a host program. There are C and C++ bindings defined in the OpenCL standard, but there are unofficial bindings for most major languages, so you can write the host program in almost whatever language you like.</p>
<p>A work-item executing a kernel has access to four different memory regions:</p>
<ul>
<li>global memory: available to all work-items executing on a device</li>
<li>constant memory: a region of global memory that remains constant during execution of a kernel</li>
<li>local memory: memory local to a work-group; can be used for communication between work-items executing in that work-group</li>
<li>private memory: memory private to a single work-item</li>
</ul>
<p>The actual implementation of these is specific to the device, and this can have a significant impact when it comes to optimising. This is one of the drawbacks of data parallelisation on the GPU; as this makes it difficult to write cross-platform code. In addition, it’s not easily adaptable to non-numeric problems.</p>
<p>However, it’s ideal when you have large amounts of numerical data that need to be processed as GPUs are powerful data-parallel processors.</p>
<h2 id="week-seven-the-lambda-architecture">Week Seven: The Lambda Architecture</h2>
<p>GPGPU is data parallelism in the small, i.e. on one computer. The last chapter looks at data parallelisation in the large, i.e. across multiple machines using the Lambda Architecture; and specifically, the batch layer and the speed layer of MapReduce on Hadoop.</p>
<p>MapReduce can mean the common pattern of breaking an algorithm down into two steps: a map over a data structure, followed by a reduce operation. However, it can also mean something more specific: a system that takes an algorithm encoded as a map followed by a reduce, and efficiently distributes it across a cluster of computers. It automatically partitions both the data and its processing, and continues to operate if one or more of these machines fails.</p>
<p>Hadoop is one framework for MapReduce. Its power comes from the fact that it splits data into sections, each of which is then processed by separate machines.</p>
<p>A MapReduce task is constructed from two type of component: mappers and reducers. Mappers take some input format and map it to a number of key/value pairs. Reducers then convert these pairs to the ultimate output format (normally also a set of key/value pairs).</p>
<p>The input typically comprises one or more large text files; Hadoop splits them and send each split to a single mapper. The mapper outputs a number of key/value pairs which Hadoop then sends to the reducers.</p>
<p>The key/value pairs from a single mapper are sent to multiple reducers. Which reducer receives a particular key/value pair is determined by the key. Hadoop guarantees that all pairs with the same key will be processed by the same reducer, no matter which mapper generated them. This is the shuffle phase.</p>
<p>Hadoop calls the reducer once for each key, with a list of all the values associated with it. The reducer combines these values and generates the final output.</p>
<p>Breaking a problem into a map over a data structure followed by a reduce operation makes it easy to parallelise.</p>
<p>Apart from speed, Hadoop has been build from the ground up to handle failure of nodes. It uses the Hadoop distributed file system (HDFS) by default, which is a fault-tolerant files system, that replicates data across multiple nodes; this means it avoids loss of data if one or more disks fail.</p>
<p>Because we are talking about gigabytes or more of data, we are at the point at which it is unreasonable to expect we’ll be able to fit intermediate data or results in memory. Hadoop stores key/value pairs within HDFS during processing, allowing us to create jobs that process much larger datasets than will fit in memory.</p>
<p>Taken together, these aspects are transformative, and what allows Hadoop to really handle big data. Throughout the book, we have kept returning to an example of counting words on pages in Wikipedia; MapReduce is the only technology that has allowed us to actually process the entire dump of wikipedia.</p>
<h3 id="raw-data-is-eternally-true">Raw data is eternally true</h3>
<p>We can divide information into two categories: raw data and derived information. Raw data is eternally true. So, for example, a home address with a timestamp is data; where you live now is derived information that might change.</p>
<p>If you had an infinitely fast computer then you would only ever store raw data because you could process it however you liked in an instant, and you wouldn’t need locking or transactions, because once it’s been stored it will never change. This is a fantasy but we can get quite close by leveraging the power of MapReduce.</p>
<p>If we know ahead of time which queries we want to run against our raw data, we can precompute a batch view which either directly contains the derived information that will be returned by those queries, or contains data that can easily be combined to create it. Computing these batch views is the job of the Lambda Architecture’s batch layer.</p>
<p>For example, if we wanted a list of my Git commits per day over a year. We don’t need to store all of them. Maybe we can precompute two totals: total per day and total per month. We can then sum as required, and if not quite start/end of month we can offset by day.</p>
<p>Because it only ever operates on raw data, data can be distributed across machines. This means the batch view can be recomputed in a reasonable amount of time even with terabytes of data. Raw data also means the system is hardened against technical failure and human error; its much easier to back up raw data but also if there’s a bug, we can always just fix the bug and recompute the batch view.</p>
<p>Note: the Lambda Architecture is not tied to MapReduce; the batch layer could be implemented with any batch processing system. He recommends Apache Spark as particularly interesting.</p>
<p>There is one problem: latency. The batch layer could take an hour to run, so your data is always an hour out of date. Hence, the speed layer.</p>
<h3 id="the-speed-layer">The speed layer</h3>
<p>As new data arrives, we both append it to the raw data that the batch layer works on and also send it to the speed layer. The speed layer generates real-time views, which are combined with the batch views to create fully up-to-date answers to queries. Real time views only contain information derived from the data that arrived since the batch views were last generated, and are discarded when their data is processed by the batch layer.</p>
<p>One way to build a speed layer is as a traditional synchronous database. In fact, you could think of a traditional database application as a case of the Lambda Architecture in which the batch layer never runs. In this approach, clients communicate directly with the database and block while it’s processing each update.</p>
<p>Another approach is asynch: clients add updates to a queue (e.g. Kafka) as they arrive and without blocking, and a stream processer then handles these updates in turn and performs the database update. Using a queue decouples clients from database updates, making it more complex to coordinate updates. This might be acceptable for many applications, and where it is, there are a lot of benefits. No blocking means higher throughput. Asynch processing means that spikes in demand just make it fall behind rather than time out or drop updates. Also, the stream processor can exploit parallelism.</p>
<h3 id="ping-pong">Ping pong</h3>
<p>Side note: the speed layer needs to have capacity for twice the amount of time data as the batch is behind, e.g. if the batch later takes one hour to run, the speed layer will need up to two hours, because if the batch layer has just finished, the data is one hour out of date. So the speed layer needs to serve data for the hour it was running, plus any data that comes in while the next one is running.</p>
<p>One common way to handle this is to have two speed layers. Whenever a batch run completes and new data becomes available in the batch views, we switch to the other speed layer, and the one we switched from clears its database and starts building a new set of views – so we never have to worry about which data to delete. This is called ping-pong speed layers.</p>
<h3 id="the-lambda-architecture-ties-it-all-together">The Lambda Architecture ties it all together</h3>
<p>The Lambda Architecture brings together many of the concepts from the book, e.g.:</p>
<ul>
<li>Raw data being eternally true is reminiscent of Clojure’s separation of identity and state</li>
<li>Hadoop’s approach of parallelising a problem by splitting it into a map over a data structure followed by a reduce is reminiscent of parallel functional programming</li>
<li>Like Actors, the Lambda Architecture distributes processing over a cluster to improve performance and provide fault tolerance</li>
</ul>
<p>The main weakness of the Lambda Architecture is that if your data is not measured in tens of gigabytes or more, the overhead is unlikely to be worth the benefit.</p>
<p>However, it is a fitting end to the book as it’s a powerful demonstration of how parallelism and concurrency allow us to tackle problems that would otherwise be intractable.</p>
<h3 id="not-everything-was-covered">Not everything was covered</h3>
<p>The author lists some other techniques that aren’t covered in the book, e.g.:</p>
<ul>
<li>Fork/join and work-stealing, e.g. Cilk</li>
<li>dataflow in much detail, mainly because there isn’t a good general purpose dataflow language</li>
<li>Reactive programming, e.g. Rx</li>
<li>Functional reactive programming, e.g. Elm</li>
<li>Grid computing, e.g SETI@Home</li>
<li>Tuple spaces, e.g. Linda</li>
</ul>
<h2 id="the-authors-summary-of-the-book-immutability-is-key">The author’s summary of the book: immutability is key</h2>
<p>The author’s summary is that the central concept is immutability, because it helps with everything, e.g. raw data, fewer locks etc. And it seems clear that even if you are not writing in a functional language, the frameworks you use and code you write will be increasingly influenced by functional principles, which will allow us to exploit concurrency and parallelism, as well as making code simpler, easier to understand and more reliable.</p>
<p>One reason for the interest in all this is multiple cores. Instead of individual cores becoming faster, we’re seeing CPUs with more and more cores, and soon shared memory might be a bottleneck, so we’ll have to worry about distributed memory, which he thinks makes it inevitable that techniques based on message-passing will become more important over time.</p>
<p>These thoughts were backed up by <a href="https://qconlondon.com/london2022/keynote/future-microprocessors">Sophie Wilson’s QCon keynote</a>. For example, she pointed out the inverse of Moore’s law is that you now need <a href="https://acquisitiontalk.com/2020/08/more-on-the-slowdown-in-science-and-technology/">18 times as many scientists working on holding performance constant</a> than the 1970s. General purpose performance hasn’t increased, though it has got more energy efficient – but it costs more. It’s now about packaging, rather than getting bigger.</p>
<h2 id="my-summary-of-the-book-extremely-interesting">My summary of the book: extremely interesting</h2>
<p>I started reading this on 11th June and finished on 20th October 2021, so not quite seven weeks; in fact it took 18 weeks and 5 days to read, plus a lot more time parsing my notes enough to write this blog post (and then a lot more editing it – the first draft was 15,000 words!)</p>
<p>The book was extremely interesting and challenging. To begin with, I did all the code exercises, and read all the supporting documentation, and really enjoyed the brain exercise: as I no longer write code or have to understand programming documentation in my day to day work, it was fun exercising my brain. For example, I really enjoyed reading <a href="https://clojure.org/reference/special_forms#binding-forms">some Clojure documentation</a> several times until the lightbulb suddenly came on, and learning new words, like “arity”. It made me feel smarter, and reinforced some things I already knew.</p>
<p>However, as I got deeper into the book and time marched on, I ultimately stopped doing all the exercises, as it wasn’t clear that I’d ever be able to finish the book at that rate. However, this is definitely a book you could dip into if you wanted more background on any of the areas: it’s incredibly detailed and well thought out.</p>
<p>The book was published in 2014, so it’s a snapshot in time, but it’s still extremely useful to to have this fascinating deeper dive into some areas I knew about but certainly didn’t understand the details.</p>
<p>I highly recommend it!</p>
Being a Principal Engineer in Customer Products at the FT2022-06-27T00:00:00+00:00https://www.annashipman.co.uk/jfdi/being-a-principal-at-FT.html<p>At the FT we have a role called Principal Engineer. Many companies do; however, the shape of the role at the FT is different than it is at many other companies, so in this post I explain what the role is like on my team, Customer Products.</p>
<p>And if you are interested, we are hiring a <a href="https://boards.eu.greenhouse.io/financialtimes33/jobs/4053976101">people-focused Principal Engineer</a> right now!</p>
<h2 id="we-have-durable-multi-disciplinary-teams">We have durable, multi-disciplinary teams</h2>
<p>The Customer Products group is responsible for the FT.com website and our iOS and Android apps. We have 11 <a href="https://medium.com/ft-product-technology/unlocking-value-with-durable-teams-a70efb435a19">teams formed around product areas</a>. Each of our teams has a <a href="https://medium.com/ft-product-technology/technical-leadership-in-customer-products-cec00bf983bd">tech lead</a>, a product manager and a delivery manager, who together are responsible for setting the direction for the team, working out what opportunities to focus on, and weighing up priorities within the team. Each team also has a number of engineers, and if relevant to that team’s work, design, user research and data roles.</p>
<p>There are currently five principal engineers on Customer Products. They report to me, the Tech Director, and they each oversee two or three product teams.</p>
<h2 id="a-principal-engineer-does-technical-and-people-focused-work">A Principal Engineer does technical and people-focused work</h2>
<p>Each Principal Engineer is like a Tech Director of their own area. That means they oversee the technical direction of their teams and support the tech leads where necessary; they each line manage around three senior engineers; and they work with me on the <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">tech strategy</a>.</p>
<p>Principal Engineers also work on strategic initiatives that reach out of Customer Products, with other tech groups, or the rest of the business. These might be technical, for example working with other groups to improve how we do AWS key rotation, or they might be people focused, for example building a career competency framework.</p>
<p>The diagram below shows some of the kinds of work that a Customer Products principal engineer might work on.</p>
<p><img src="/img/principal_engineer_responsibilities.png" alt="A diagram listing some of the things principal engineers might work on, ranging from technical on the left (e.g. “strategic tech projects”) to managerial on the right (e.g. “leading recruitment”)" /></p>
<h2 id="i-want-the-principal-engineers-to-be-ready-for-my-job">I want the Principal Engineers to be ready for my job</h2>
<p>My goal is for the Customer Products Principal Engineers to be running Customer Products engineering without me, and for any one of them to be a good candidate to take my job, as Tech Director for Customer Products.</p>
<p>Some of the things we’ve done towards that include <a href="https://www.annashipman.co.uk/jfdi/delegating-to-a-team.html">clarifying the different types of work we do</a> and making sure that we share as much context as possible, both from me to the team and the team with each other and me. We also follow an agile planning process and try to <a href="https://www.annashipman.co.uk/jfdi/prioritising-in-tech-leadership.html">focus on the ‘rocks’ rather than the ‘pebbles’ and ‘sand’</a>.</p>
<p>However, we are currently in the position that although the Customer Products Principals excel in both technical and people areas, the current preferences of the team are more skewed towards the technical end of the spectrum in the diagram above.</p>
<h2 id="we-are-hiring-a-people-focused-principal-engineer">We are hiring a people-focused Principal Engineer</h2>
<p>With any team you want a complementary mix of skills, so to create a more balanced senior team we’re hiring a people-focused Principal Engineer.</p>
<p>The kind of things that a people-focused Principal Engineer might lead on include:</p>
<ul>
<li><a href="https://medium.com/ft-product-technology/unlocking-value-with-durable-teams-a70efb435a19">Leading the move to durable teams</a></li>
<li><a href="https://medium.com/ft-product-technology/improving-our-career-map-for-engineers-4210185c6246">Building a career competency framework</a></li>
<li><a href="https://medium.com/ft-product-technology/making-recruitment-fair-for-people-of-colour-66a3ad907a7d">Improving our recruitment process to make it fairer for people of colour</a></li>
</ul>
<p>It is also an opportunity for someone to lead on the more people-focused aspects of leadership within Customer Products, to gain experience to be ready for that part of the Tech Director role; for example ensuring our retention and reward processes are fair, effective and work within our budget, or looking at ways to improve the experience of working at the FT and in Customer Products; whether that is to do with productivity, diversity and inclusion, technical competence or other areas.</p>
<p>However, this is not a purely people and process role. This person will still have oversight of the technical direction of two to three teams, support the tech leads in making well-informed decisions, and work with other principal engineers and other tech groups.</p>
<p>The difference between this role and a more technical-focused principal engineer role will be in the types of strategic project they initiate and lead.</p>
<h2 id="we-do-not-currently-have-engineering-managers-at-the-ft">We do not currently have Engineering Managers at the FT</h2>
<p>The people-focused Principal Engineer role I’ve described above is very similar to what many organisations call Engineering Managers. However, we do not have this role at the FT at the moment.</p>
<p>Before I joined the FT, there were Engineering Managers. At that time, the organisation was very siloed by discipline, and as part of a move towards multidisciplinary teams, it was decided that this role was no longer needed. Much of the work that Engineering Managers did then is done by Delivery Managers now, and the role then was quite different to the role we would describe as an Engineering Manager now.</p>
<p>At the FT, each tech group is set up in a slightly different way. We want ultimately to align our structures; so we might create an Engineering Manager role in the future, but that is a much bigger piece of work across the whole of technology.</p>
<h2 id="join-us">Join us!</h2>
<p>The FT is a great place to work. My colleagues are smart, motivated and kind, and our purpose is extremely important: it is hard to overstate the value of balanced and unbiased journalism in today’s world.</p>
<p>We use modern engineering practices and are always looking for ways to improve, and the organisation is one that cares about its staff. If this sounds like a place you’d like to work then please <a href="https://boards.eu.greenhouse.io/financialtimes33/jobs/4053976101">apply for this role</a>, or check out our <a href="https://boards.eu.greenhouse.io/financialtimes33/">many other open vacancies</a>.</p>
<p><em>This post originally appeared on the <a href="https://medium.com/ft-product-technology/being-a-principal-engineer-in-customer-products-at-the-ft-97ac3ecc1c9d">FT Product & Technology blog</a></em></p>
The difficult teenage years: setting your tech strategy after the launch2022-06-11T00:00:00+00:00https://www.annashipman.co.uk/jfdi/difficult-teenage-years-leadeng.html<p>I delivered a talk at the inaugural London LeadingEng conference on <a href="https://leaddev.com/technical-direction-strategy/difficult-teenage-years-setting-your-tech-strategy-after-launch">how to set a tech strategy, specifically after the launch</a>.</p>
<p>This was an updated version of my <a href="/jfdi/after-the-launch.html">talk from Continous Lifecycle London in 2019</a>. It was planned to include learnings one year on, but the original LeadingEng was scheduled for summer 2020, so by the time we had the conference, I was able to include learnings three years on from setting the strategy.</p>
<p>Slides below, and the <a href="https://vimeo.com/720550390/123b9a671b">video is on Vimeo</a>.</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/zh7E57Nt7gavld" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe>
No Next Next: Fighting Entropy in Your Microservices Architecture2022-04-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/fighting-entropy-qcon.html<p>On Tuesday, I delivered a talk at QCon London: <a href="https://qconlondon.com/london2022/presentation/no-next-next-fighting-entropy-microservices-architecture">No Next Next: Fighting Entropy in Your Microservices Architecture</a>.</p>
<p>This was my first in person conference talk since 2019, and I really enjoyed it.</p>
<p>I got some great questions after the talk, including questions about:</p>
<ul>
<li>how we set up durable teams (you can read more in the blog post <a href="https://medium.com/ft-product-technology/unlocking-value-with-durable-teams-a70efb435a19">Unlocking value with durable teams</a>);</li>
<li>how we handled teams suddenly having to own a lot of tech they didn’t know about (it takes time, and is still a work in progress; it’s much better if you don’t get into a situation where you have unowned tech in the first place);</li>
<li>how to sell technical work to the business (<a href="https://www.infoq.com/articles/communicating-engineering-work-business/">Talking like a suit</a> is an excellent article on that topic)</li>
<li>and how to work on a tech strategy when there isn’t a clear business or product strategy (I think that may be a whole other talk in itself!)</li>
</ul>
<p>I also got asked a very interesting question about how to improve your storytelling. As I’d mentioned a couple of times in my talk, telling a good story is really important to communication. I didn’t immediately have an answer, but I put it to Twitter – check out the thread for a lot of great suggestions.</p>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">If anyone else has ideas about how to improve your storytelling, please chip in!</p>— Anna Shipman (@annashipman) <a href="https://twitter.com/annashipman/status/1511699820834136066?ref_src=twsrc%5Etfw">April 6, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Slides from my talk are below. A recorded version of the talk, used in QConPlus, is <a href="https://www.infoq.com/presentations/microservices-entropy/">on the InfoQ site</a>.</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/eD9G6SNvuINGR4" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe>
<div style="margin-bottom:5px"> </div>
Encouraging meeting participation with a check-in2022-03-22T00:00:00+00:00https://www.annashipman.co.uk/jfdi/meeting-check-in.html<p>If you want participation in a meeting with people who don’t spend a lot of time together, one thing that can help is getting everyone to say something at the start; here is a low-pressure way to do that.</p>
<h2 id="speaking-up-in-a-meeting-with-new-people-requires-confidence">Speaking up in a meeting with new people requires confidence</h2>
<p>It can be hard to speak up in meetings. When you’ve worked with the same group of people for a while, it gets easier, but if you are running a meeting where there are new people, or people who don’t know each other, it’s worth having a way to make it a little bit easier for people to feel able to speak up.</p>
<h2 id="its-good-to-have-have-something-low-importance-to-say-first">It’s good to have have something low importance to say first</h2>
<p>Any of you who have met me in person might be surprised to hear that I used to be extremely shy. I did speak up in meetings, but I would have to spend time gearing myself up to do so, and I also used to blush whenever I spoke.</p>
<p>When you are not as confident speaking up in a meeting, you can spend a lot of mental energy wondering about <em>how</em> to say what you want to say, including things like how loud your voice will have to be, and often the moment will have passed. Once you’ve made your first contribution though, the next one is easer as the technicalities are out of the way.</p>
<p>So in a meeting where you want everyone to participate, it’s best to give everyone a chance to practise using their voice on a low effort, low impact comment at the beginning,</p>
<h2 id="dont-make-it-something-meaningful">Don’t make it something meaningful</h2>
<p>The first time I was involved in a retrospective at work, many years ago, the facilitiator asked us each to warm up with a comment about “our hopes and fears for the retrospective”. That’s a pretty hard question! First you have to think of the answer, and then you have to filter it for what you think is appropriate to say in that context with those people (“I fear this is going to be the biggest waste of my time” is probably not the opening gambit you want to go with.)</p>
<p>Personal questions are also tricky. Some people like sharing at work, and some people do not; so a question that involves revealing something about yourself can also be pretty stressful. (“Tell us about the best holiday you’ve ever had”, or “If you were an animal, what animal would you be and why” are examples of questions I do not want to answer unless I’m comfortable already in that team.)</p>
<p>I favour questions that are unrevealing if the person wants them to be; brief; and something that just gets participants to practise speaking and being heard in the meeting.</p>
<h2 id="some-sample-questions">Some sample questions</h2>
<p>I usually offer a choice of questions so that people can choose what they want to share. The last time I used this, I asked for one of the following:</p>
<ul>
<li>Recommend a TV show</li>
<li>What are you reading?</li>
<li>What colour are your socks</li>
<li>…or anything else! (brief and SFW*)</li>
</ul>
<p>*Safe For Work.</p>
<h2 id="some-more-imaginative-questions--but-be-careful-with-these">Some more imaginative questions – but be careful with these</h2>
<p>If you are doing this regularly with the same team, and people are expecting it, it can be fun to play around a bit with the questions, e.g. basing them on current events:</p>
<ul>
<li>If you were at the Cheltenham race meeting today, how would you rate…?</li>
<li>If you were a Eurovision song contest judge, how many points would you give…?</li>
<li>If you were a Crufts dog show judge, what breed would you consider…?</li>
<li>If you were a traffic warden, how much would you fine….?</li>
<li>If you were an interviewer on the Today programme, what would you ask…?</li>
</ul>
<p>Be careful with these ones though; they can put new people on the spot in an unhelpful way, and some of them are not appropriate for your workplace.</p>
<p>For example the last one is only appropriate in a workplace where <em>everyone</em> feels comfortable discussing politics. If you’re not 100% sure that absolutely everyone is comfortable with that, stay away.</p>
<h2 id="keep-them-simple">Keep them simple</h2>
<p>The purpose of the check-in is solely that everyone has a chance to practise speaking in that meeting and being heard.</p>
<p>So it doesn’t matter what is said, just that they’ve already spoken, before the time comes where they may have an important contribution to make.</p>
<h2 id="this-is-not-a-silver-bullet">This is not a silver bullet</h2>
<p>This is not the only thing you have to do to encourage participation, and on its own, it won’t get people to speak up who do not feel inclined to do so. For that, you need to also do other things: smaller groups, deliberate turn-taking, doing the work to build and maintain psychological safety, and much more.</p>
<p>However, this is one small way you can help shy people like past Anna.</p>
The good, the bad and the tech strategy2022-02-16T00:00:00+00:00https://www.annashipman.co.uk/jfdi/good-bad-tech-strategy.html<p>Yesterday, I delivered a talk at <a href="https://events.leaddev.com/together">LeadDev Together</a>: The good, the bad and the tech strategy.</p>
<p>I covered what strategy is and what makes a good strategy (both very much distillations of the excellent <a href="/jfdi/good-strategy-bad-strategy.html">Good Strategy, Bad Strategy</a>), and then I talked about what a good tech strategy looks like.</p>
<p>My slides are below and I will post a link to video when I have it.</p>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/kheCNvLct8Hk5u" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe>
<div style="margin-bottom:5px"> </div>
Why didn't Clojure work concurrently in Vagrant?2021-12-31T00:00:00+00:00https://www.annashipman.co.uk/jfdi/clojure-vagrant.html<p>I recently couldn’t work out why Clojure was not working concurrently in a Vagrant box. I didn’t solve the problem, but I learned many things, and here are some of them.</p>
<h2 id="i-use-vagrant-for-most-things-i-do">I use Vagrant for most things I do</h2>
<p>I love <a href="https://www.vagrantup.com/">Vagrant</a>. I love being able to put things down and pick them up where I left off, without having to recreate environments. I do not have a good memory, and the Vagrantfile is built-in documentation. I create a Vagrantfile for almost every side project I do.</p>
<p>Of course, this means I sometimes (or often…) end up going down a rabbithole that has very little to do with the actual side project itself. However, these are usually most entertaining, and I learn things. Here is one rabbithole and what I learned.</p>
<h2 id="i-could-not-replicate-a-concurrency-gain">I could not replicate a concurrency gain</h2>
<p>I was working my way through <a href="https://pragprog.com/titles/pb7con/seven-concurrency-models-in-seven-weeks/">Seven Concurrency Models in Seven Weeks</a> and running some code examples on my Vagrant machine, and I found that a promised speed-up by using a function that supported concurrency did not materialise.</p>
<p>In a <a href="https://leiningen.org/">Leiningen</a> REPL I ran:</p>
<div class="code-sample">
<p>(ns sum.core<br />
(:require [clojure.core.reducers :as r]))</p>
<p>(defn sum [numbers]<br />
(reduce + numbers))</p>
<p>(defn parallel-sum [numbers]<br />
(r/fold + numbers))</p>
<p>(def numbers (into [] (range 0 10000000)))</p>
<p>(time (sum numbers))</p>
<p>(time (sum numbers))</p>
<p>(time (parallel-sum numbers))</p>
<p>(time (parallel-sum numbers))</p>
</div>
<p>That is, I define a function called <code class="language-plaintext highlighter-rouge">sum</code> which uses <code class="language-plaintext highlighter-rouge">reduce</code>, and one called <code class="language-plaintext highlighter-rouge">parallel-sum</code> that uses <code class="language-plaintext highlighter-rouge">fold</code>. I then use <code class="language-plaintext highlighter-rouge">time</code> to see how long each function takes to sum a range of numbers from 0 to 10,000,000. To quote the book, we run each of the timing functions twice “to give the just-in-time optimizer a chance to kick in and get a representative time”.</p>
<p><a href="https://clojuredocs.org/clojure.core/reduce"><code class="language-plaintext highlighter-rouge">reduce</code></a> is not parallel, but <a href="https://clojuredocs.org/clojure.core.reducers/fold"><code class="language-plaintext highlighter-rouge">fold</code></a> can be, and according to the author, on his four-core Mac, the <code class="language-plaintext highlighter-rouge">fold</code> function gave around a 2.5x speed-up.</p>
<p>However, on my Vagrant box, the two functions ran in approximately the same length of time.</p>
<h2 id="i-eventually-asked-for-help-on-twitter">I eventually asked for help on Twitter</h2>
<p>After quite a bit of investigation myself, including <a href="https://github.com/annashipman/7weeks-concurrency/commit/360bedc">correcting some foolish errors</a>, I was not able to work out why this wasn’t working. Vagrant could see all four of the cores (<code class="language-plaintext highlighter-rouge">(.availableProcessors (Runtime/getRuntime))</code> showed it could see 4) but the allegedly parallel code was no faster.</p>
<p>I figured this must be something to do with Vagrant, and I was committed to getting it all working on my Vagrant machine, if I could.</p>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">Do I know anyone who knows Vagrant and Clojure and can help me debug a (minor) concurrency issue?</p>— Anna Shipman (@annashipman) <a href="https://twitter.com/annashipman/status/1439527358701113346?ref_src=twsrc%5Etfw">September 19, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>On <a href="https://twitter.com/borkdude/status/1439529936218435587">@borkdude</a>’s suggestion I <a href="https://gist.github.com/annashipman/d3b0533ce26df1e4dd84fbc3001e98dc">wrote it up in a gist</a>.</p>
<p>I got some very helpful responses.</p>
<h2 id="it-could-be-how-virtualbox-creates-cpus">It could be how VirtualBox creates CPUs</h2>
<blockquote class="twitter-tweet" data-conversation="none" data-dnt="true"><p lang="en" dir="ltr">I don't know clojure but based on your info I would guess it might be how VirtualBox creates cpus.<br /><br />4 cores on a single cpu share L3 cache but 4 cpus with one core doesn't. That might make enough difference for clojure.core.reducers to determine it's not worth doing in parallel</p>— Pär Björklund (@Paxxi) <a href="https://twitter.com/Paxxi/status/1439650030844063746?ref_src=twsrc%5Etfw">September 19, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Until I read this tweet, I had not consciously realised that CPUs and cores were different things. Sometimes the two terms are used interchangeably – incorrectly, I now know.</p>
<p>A CPU is made up of one or more cores, an input and output management unit, and some other things. The cores are the parts that actually do the work of executing instructions, so multiple cores allow the CPU (in theory at least) to carry out multiple instructions at once. A dual-core machine is not a computer with two CPUs, it’s a computer with one CPU that has two cores. It is also possible for computers to have multiple CPUs, either single-core or multi-core.</p>
<p>CPUs have caches to speed up access to memory; L1, L2 and L3. The details vary between manufacturers and chips, but the L3 cache is the biggest CPU cache and is usually built between the CPU and the RAM. Each core usually has its own L1 cache and possibly L2 cache, though L2 may be shared. L3 cache is usually shared between the cores.</p>
<p>So Pär Björklund’s suggestion was that if VirtualBox created the multiple CPUs as CPUs rather than cores, the L3 cache wouldn’t be shared, so parallelising would not produce much of an efficiency gain, and therefore Clojure may decide not to do it.</p>
<p>The investigation into whether my virtual machine has multiple cores or multiple CPUs involved much unpicking. <a href="https://www.vagrantup.com/docs/providers/virtualbox/configuration">Vagrant’s docs</a> refer to setting CPUs. It links to the <a href="https://www.virtualbox.org/manual/ch08.html">VirtualBox manual</a> which refers also to CPUs (“<code class="language-plaintext highlighter-rouge">--cpus <cpucount></code>: Sets the number of virtual CPUs for the virtual machine”) but refers you on to <a href="https://www.virtualbox.org/manual/ch03.html#settings-processor">another part of the manual</a> which reveals that it’s actually CPU <em>cores</em>: “Sets the number of virtual CPU cores the guest OSes can see.”</p>
<p>From this, I learned that <a href="https://github.com/annashipman/7weeks-concurrency/commit/2e81a52">I should not have set my guest VM to have 4 CPUs on my dual-core host</a>, though not unexpectedly, making that change did not affect the speed of the <code class="language-plaintext highlighter-rouge">parallel-sum</code> function.</p>
<p>I did establish that it looks like VirtualBox is creating multiple cores rather than multiple CPUs, so in theory they may be sharing L3 cache, though I can’t say definitively whether they are. In any case, it doesn’t seem that I have any control over it.</p>
<p>Digging into this suggestion led me to some very interesting learnings about the structure of computers, and might actually be the root of the problem, but even if it is, it’s not clear if I can solve it. So I moved on to other suggestions.</p>
<h2 id="it-could-be-that-there-are-not-enough-threads">It could be that there are not enough threads</h2>
<p><a href="https://twitter.com/d_a_keldsen">Dak</a> asked what I had set the virtual CPUs and CPU cap to:</p>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">I’m thinking that the default of 50% may be restricting the creation of additional threads. Also, try adding something that causes a kernel call to force another thread to run.</p>— Dak (David A. Keldsen) (@d_a_keldsen) <a href="https://twitter.com/d_a_keldsen/status/1439963498734395399?ref_src=twsrc%5Etfw">September 20, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>The CPU execution cap controls how much of the host’s CPU time a virtual CPU can use. The <a href="https://www.virtualbox.org/manual/ch03.html#settings-processor">VirtualBox documentation</a> doesn’t appear to say what the default is, though the suggestion above is that its 50%, which makes sense.</p>
<p>However, explicitly setting it to 50% and then 100% on my VM made no difference. At 100% it was faster, but there was still no difference between the speed of <code class="language-plaintext highlighter-rouge">reduce</code> and <code class="language-plaintext highlighter-rouge">fold</code>.</p>
<p>The idea of making a kernel call is to call an interrupt to test whether control can pass from thread to thread. A kernel interrupt would be higher priority than my Clojure function; so if I called a kernel interrupt and it wasn’t successful, I would know threading is not working as expected in my VM.</p>
<p>I wasn’t sure how to do this. In Java, I could call the <code class="language-plaintext highlighter-rouge">interrupt()</code> method of <code class="language-plaintext highlighter-rouge">Thread</code>, but in Clojure, all the threading is being done under the hood. However, in any case, we know that there is enough CPU being allowed for the guest to create new threads because some of the other threading in the book does work. Relevantly, the Java threading examples work, and Clojure runs on the JVM.</p>
<p>Even though it looks like this is not the cause of the problem, it’s a very interesting suggestion and helped consolidate my understanding of how threading works.</p>
<h2 id="it-could-be-to-do-with-the-jvm-garbage-collection">It could be to do with the JVM Garbage Collection</h2>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">LEIN_JVM_OPTS="-verbose:gc -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=50 -XX:+UseG1GC -Xms3g -Xmx3g" lein repl</p>— Philip Wigg (@philipwigg) <a href="https://twitter.com/philipwigg/status/1439693698539864067?ref_src=twsrc%5Etfw">September 19, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p><a href="https://twitter.com/philipwigg">Philip Wigg</a> had the answer that had the most impact. He suggested that it might be the JVM Garbage Collection that was messing with the results.</p>
<p>Instead of starting the Leiningen REPL with <code class="language-plaintext highlighter-rouge">lein repl</code>, start it with <code class="language-plaintext highlighter-rouge">LEIN_JVM_OPTS="-verbose:gc -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=50 -XX:+UseG1GC -Xms3g -Xmx3g" lein repl</code>.</p>
<p>To unpick this:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">LEIN_JVM_OPTS</code>: <a href="https://github.com/technomancy/leiningen/blob/master/doc/TUTORIAL.md#setting-jvm-options">set options for the leiningen JVM</a></li>
<li><code class="language-plaintext highlighter-rouge">verbose:gc</code>: make the garbage collector verbose</li>
<li><code class="language-plaintext highlighter-rouge">XX:+UnlockExperimentalVMOptions</code>: the version of Java on my VM is Java 8, so the following are all experimental at that stage</li>
<li><code class="language-plaintext highlighter-rouge">XX:G1NewSizePercent=50</code>: set the percentage of the heap to use as the minimum for the young generation size</li>
<li><code class="language-plaintext highlighter-rouge">XX:+UseG1GC</code>: explicitly use the <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/g1_gc.html#garbage_first_garbage_collection">Garbage First Garbage Collector</a>. This is now the default, but was not for Java 8</li>
<li><code class="language-plaintext highlighter-rouge">Xms3g -Xmx3g</code>: set the initial and max heap size</li>
</ul>
<p>While looking into this I found this <a href="https://technospace.medium.com/gc-allocation-failures-42c68e8e5e04">very useful and interesting article about garbage collection</a>, most of which I’d forgotten about since my time as a Java programmer, many years ago.</p>
<p>The theory here is that on the host, the JVM has access to more RAM, but on the smaller VM, without this tuning, it has to keep doing garbage collection, which slows things down.</p>
<p>This did make <code class="language-plaintext highlighter-rouge">parallel-sum</code> faster for me. And here are his results:</p>
<blockquote class="twitter-tweet" data-conversation="none" data-dnt="true"><p lang="en" dir="ltr">Slightly modified version of your script that runs 10 of each. Couple of slow ones due to GC but otherwise Fold wins. <a href="https://t.co/Xy4hJAF8qV">pic.twitter.com/Xy4hJAF8qV</a></p>— Philip Wigg (@philipwigg) <a href="https://twitter.com/philipwigg/status/1439696075980525571?ref_src=twsrc%5Etfw">September 19, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>However, I never got to the promised 2.5x speed-up, and I later found that this did not even consistently speed up my results. But once again, it was very entertaining and I learned (and remembered!) some interesting things.</p>
<h2 id="i-never-got-to-the-bottom-of-it">I never got to the bottom of it!</h2>
<p>This was a very interesting digression. I learned a lot that I didn’t know, some of it about concurrency, and some about how computers work, always a fascinating topic.</p>
<p>Of course, I didn’t actually solve my problem! An easy way to have replicated these concurrency gains would have been to install Clojure and Leiningen on my Mac and run it like that. But one of the other reasons I like Vagrant is that I don’t like to install things on my computer that I don’t need. I don’t even have Java on this computer. So installing all that on my computer would have taken away another of the main benefits of using Vagrant for me.</p>
<p>If you want to follow along the code examples in the <em>Seven concurrency models</em> book, you can use <a href="https://github.com/annashipman/7weeks-concurrency/blob/main/Vagrantfile">my Vagrantfile</a> to run some of the examples. But be warned, the Clojure ones do not all work as expected. Hopefully you now have some hints about why!</p>
<h2 id="edit-it-could-be-a-problem-with-the-jdk-that-virtualbox-installs">Edit: It could be a problem with the JDK that VirtualBox installs</h2>
<p>After publishing this blog post, <a href="https://twitter.com/linusericsson">Linus Ericsson</a> suggested another theory:</p>
<blockquote class="twitter-tweet" data-conversation="none" data-dnt="true"><p lang="en" dir="ltr">One problem I had with JDK 8 before 181 or so running in *Docker* was that they did not honour Dockers way of reporting the number of CPU:s (using cgroups). This lead to the JVM:s go above the Docker:s quotas and being terminated or fail.<a href="https://t.co/PAIFPjkpGD">https://t.co/PAIFPjkpGD</a></p>— Linus Ericsson (@linusericsson) <a href="https://twitter.com/linusericsson/status/1478096719762137100?ref_src=twsrc%5Etfw">January 3, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr">My theory is that the old JDK version misinterprets the amount of resources (cores, perhaps memort) that are made availiable in the Vagrant virtual box and the program therefore make incorrect decisions when spawning the fork-join-processes.</p>— Linus Ericsson (@linusericsson) <a href="https://twitter.com/linusericsson/status/1478101205469515778?ref_src=twsrc%5Etfw">January 3, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>This is very plausible. Unfortunately, the default JDK for the OS I’m running is 1.8.0_292, and the latest available is 1.8.0_312: not a big difference, and installing that did not solve my problem. So again, this could be the solution but I cannot fix it if so.</p>
Working on the most important things as a technical leadership team2021-09-28T00:00:00+00:00https://www.annashipman.co.uk/jfdi/prioritising-in-tech-leadership.html<p>Two years ago, I wrote about how as a technical leadership team we can <a href="/jfdi/delegating-to-a-team.html">figure out what to work on</a>. We made some changes to the way we worked and found them really beneficial. However, last year we realised there were flaws in how we were working, which made it hard for us to prioritise the bigger, more impactful pieces of tech work.</p>
<p>Here are the changes we made to address that. We still have some way to go and would love your thoughts.</p>
<h2 id="the-pandemic-had-knocked-out-our-processes">The pandemic had knocked out our processes</h2>
<p>One reason we had less time for larger pieces of tech work was that over the course of 2020, team health work became more important. We all found ourselves doing (and needing!) more of that, both due to the external stresses of the pandemic, and internal pressures around changes that had to be made because of it.</p>
<p>In addition, when the pandemic hit and we all moved to working from home, a few of our processes also dropped out. For example, we no longer had a physical wall, and in the rush and emotional turmoil of those first few working from home weeks, we dropped having a stand-up meeting and did not restart it.</p>
<h2 id="individual-principal-engineers-did-not-feel-confident-making-big-technical-decisions">Individual principal engineers did not feel confident making big technical decisions</h2>
<p>By the end of summer 2020, all of the principal engineers and I felt that we had been struggling to make time for the larger pieces of technical work.</p>
<p>However, it wasn’t just about making time. It emerged that none of the principal engineers felt empowered to push forward on any individual piece of tech work, because all felt uncertain about what the priorities were or the direction of travel.</p>
<p>They all felt confident in making decisions about team health, recruitment etc, often without discussion with each other, but with technical decisions there was uncertainty about the overall direction and what the priorities were, which meant that only small technical issues got worked on, and mostly in a very reactive way.</p>
<p>We all felt that we were avoiding the bigger technical issues, and spending too much time on lower value work, but didn’t understand why or how to fix it.</p>
<h2 id="we-had-prioritised-tech-work-using-a-card-exercise">We had prioritised tech work using a card exercise</h2>
<p>I’ve <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">written before about how we initially prioritised work on our tech strategy</a>. One of the most useful exercises we did for prioritisation was a card exercise where we laid cards out on the floor in the order we felt work needed to be done and then walked the room discussing dependencies and things we hadn’t considered. The participants were senior engineers, principal engineers, the product director and myself.</p>
<p>This exercise was extremely useful for:</p>
<ul>
<li>Pulling out the big areas that needed attention</li>
<li>Exploring context with all those present so we had a shared understanding of the challenge, dependencies and future direction</li>
<li>Outlining the beginning of a roadmap</li>
</ul>
<p>We left that meeting with an idea of the top three priorities to work on. We then had shorter, quarterly meetings, where we clarified where we were with the top three priorities, and when work was finished, identified the next priority or priorities.</p>
<p>The initial card-laying session, supplemented with the quarterly discussions, despite changes in personnel of a lot of people in the roles, saw us through 18 months of understanding what our technical direction was and clarity on what the next steps were to get there.</p>
<h2 id="we-could-not-do-the-card-exercise-remotely">We could not do the card exercise remotely</h2>
<p>By the beginning of 2020, it was clear that we were getting to the end of the roadmap we’d defined, and we did not have alignment among ourselves, either within the principal engineers or more widely within engineering, as to what the next priorities were.</p>
<p>So we planned another away day to go through the card-laying exercise again. This was booked in for March 26th, 2020. However, on March 16th the FT moved to working from home, and by March 23rd the UK was in lockdown.</p>
<p>We cancelled the meeting.</p>
<p>In the first, very stressful, few months of the pandemic, it didn’t feel possible to think about how to plan ahead. Once we were able to do so, we managed to find good ways to prioritise and vote remotely, but what we were missing was the bigger picture view we had hoped to get with the full day discussion and we could not work out how to do achieve the same outcomes remotely or asynchronously.</p>
<h2 id="rocks-pebbles-and-sand-to-focus-our-time">Rocks, pebbles and sand to focus our time</h2>
<p>As I described in my <a href="/jfdi/delegating-to-a-team.html">previous blog post about how we worked</a>, we felt we ought to be splitting our time 1/3 team health, 1/3 tech strategy, 1/3 tech oversight. But we were not keeping track of the time. This hadn’t turned out to be a good suggestion as the overhead for implementation was too great, so we decided to think about it another way than splitting time.</p>
<p>An analogy I’m fond of is the one about how if you put sand in a jar first you do not have room for rocks, where rocks represent life’s big priorities and sand the little things, whereas if you put the rocks in first you’ll still have room for some sand. (Here is <a href="https://shepleywood.com/news/rocks-pebbles-sand-and-beer">a link to a version which includes beer</a>).</p>
<p>It felt very much to me that we had some pebbles and a lot of sand, but we were avoiding picking any rocks, where rocks were big, meaty, technical issues that needed focus and time to solve.</p>
<p>So our suggestion was we crowd out the sand with rocks; i.e. be clear about what the rocks are and make sure we are always working on those.</p>
<h2 id="i-had-a-strong-sense-of-what-the-rocks-should-be">I had a strong sense of what the rocks should be</h2>
<p>When the team was feeling a bit lost at the end of the summer, I felt it was clear what the priorities should be.</p>
<p>However, just telling people what to do next is not part of how I lead. While it is my job to set and implement tech strategy, I don’t have all the answers and I don’t see everything. Successfully selling priorities, taking on board feedback, and considering other options, rather than just dictating what we do, leads to a better strategy, as well as more ownership from those who are actually going to be implementing it.</p>
<p>A blocker from me was that I didn’t want us to go off and define the tech strategy ourselves without involving the senior engineers.</p>
<p>However, it was clear that we were stalled and needed to move forward, so I put that aside and arranged a meeting with just the principal engineers to discuss the top priorities. Because there were fewer of us, we were able to have a shorter meeting that was manageable remotely.</p>
<p>We came out of that meeting with a shared understanding of the top three priorities, which solved the immediate problem of not knowing what we <em>should</em> be working on, though didn’t address the wider of issue of not getting into this mess again.</p>
<h2 id="as-well-as-identifying-the-rocks-we-reviewed-the-sand">As well as identifying the rocks, we reviewed the sand</h2>
<p>As well as crowding out the sand with rocks, we felt it was important to review what the sand represented, to be clear about what we shouldn’t be doing. Sand in this case represented work that seems urgent, and it’s easier for each of us independently to make a decision about what the right action is, so ends up taking too much of our time, but actually was not contributing to our priorities.</p>
<p>We have a list of the kind of activity a principal engineer does, split into three categories: tech strategy, tech oversight and team health. This is useful for job specs, letting people who are interested in joining the team know what we do, as well as clarifying for ourselves what’s in and what’s out. It’s also useful for making sure that we fairly share out <a href="https://noidea.dog/glue">the essential glue work that otherwise may not be fairly distributed</a>. However, we realised that some of the things on that list were not actually things that should be priorities for us.</p>
<p>We reviewed the list and questioned whether each one was actually something we should be doing. This was a really useful exercise as it identified some places we were spending our time wrong. For example we identified a number of tasks that were actually a delivery responsibility and were able to hand those over, and some further tasks that on reflection were not adding sufficient value to be worth our time.</p>
<h2 id="the-overriding-principle-is-to-prioritise-work-only-we-can-do">The overriding principle is to prioritise work only we can do</h2>
<p>Technical work is where we can really add value as senior technical leaders - and it is work that needs doing, that no-one else can do. We were not fulfilling our potential, or even doing our jobs properly by allowing sand to crowd that work out.</p>
<p>We agreed that this should be the overriding principle of what we should be doing – prioritise the work that only we can do, as senior technical leaders. This helps us be clear about when we should say no to things.</p>
<h2 id="we-instated-some-regular-in-depth-planning-and-discussion-meetings">We instated some regular in-depth planning and discussion meetings</h2>
<p>While our one-off conversation had made sure we were all aware of what the next priorities were and were making progress towards them, I still wanted a process that supported that happening in an ongoing way.</p>
<p>My goal is still, and always has been, looking for ways to make myself redundant in a role, and in this case, for the Customer Products principal engineers to be able to run the team without me – so anything that relies on me having to state what the priorities are is a bad sign.</p>
<p>We have a weekly principals’ meeting that anyone can add items to and we go through in priority order. There’s always a lot on the agenda, and what we found was that we often didn’t have time for in-depth discussions.</p>
<p>So we made some changes:</p>
<ul>
<li>
<p>We set up two planning meetings</p>
<p>One every six weeks, called ‘future planning’, and one every other week instead of the principals’ meeting, called ‘incremental planning’.</p>
<p>In <strong>future planning</strong>, we spend two hours discussing the bigger picture and making sure our priorities are in the right order. We use a Miro board to support this discussion, and in between times we add context. This is really useful because over time the picture builds up.</p>
<p>In <strong>incremental planning</strong>, we check in, just to make sure the work we are currently doing reflects the highest priorities as agreed in future planning.</p>
<p>This is also where we look at work that people have asked us to do and see where it fits in and how it should be prioritised; make sure we don’t each have too much WIP; hand things off to each other if appropriate and drop things if they are not a priority. It is a useful context sharing meeting to check we remain aligned.</p>
</li>
<li>
<p>We reinstated stand-up for six weeks but found it duplicated incremental planning so dropped it again.</p>
</li>
<li>
<p>We set up two weekly discussion slots of an hour each. These are for the ‘pebbles’ that need more discussion than we can cover in our weekly meeting, or for making progress together on ‘rocks’.</p>
<p>These slots are booked and available so that if we do need to have a more in depth discussion about something, we don’t need to then worry about finding the time; and if there’s nothing that we need to talk about: great, we’ve got a bit of time back to do other focused work.</p>
</li>
</ul>
<p>These changes have worked really well. Future planning means we are spending time discussing the bigger picture and making sure we are still agree on the top priorities. Incremental planning makes sure we are keeping on track. And the discussion slots are really useful – sometimes we use up both the slots in a week, either ad hoc or booked in advance, and about half the time we don’t have anything on the agenda so we get some time back.</p>
<h2 id="we-had-picked-some-big-things-to-work-on-and-a-process-rather-than-a-coherent-plan-of-action">We had picked some big things to work on and a process rather than a coherent plan of action</h2>
<p>We have worked out what the next big things are, and a process to share context between principal engineers and ensure we are focused on the rocks.</p>
<p>However, the forward planning meeting and other changes we’ve made are about context sharing and thinking together. While this is very useful, what is missing is clarity around what kind of thing should be next. What should person X, as an engineer on Customer Products, do next, in order to get us to our goal?</p>
<p>Strategy should be <a href="/jfdi/good-strategy-bad-strategy.html">diagnosis, vision and a plan of coherent action</a>. The “rocks” are steps on the way, but they do not form a full plan of action, and they do not tell us how to prioritise the next thing to work on.</p>
<h2 id="some-thoughts-on-how-we-could-clarify-the-next-steps">Some thoughts on how we could clarify the next steps</h2>
<p>One of the principal engineers pointed me at Turn the Ship Around! (which I <a href="/jfdi/turn-the-ship-around.html">wrote up here</a>). This book was great, but what I struggled to apply to a technical leadership situation was how to make sure that an engineer on the team could identify what work is next.</p>
<p>When I asked for advice on this, I got some really helpful thoughts from <a href="https://chris.swanz.net/">Chris Swan</a>. “The concept I’ve found most useful to deal with this is ‘<a href="https://en.wikipedia.org/wiki/Intent_(military)">command intent</a>’. Once you know which hill you’re going to bombard, the team know the desired outcome, which is to take the hill, and they can get on with doing that even if you’re shot by a sniper moments later.</p>
<p>Amazon’s practice of <a href="https://www.allthingsdistributed.com/2006/11/working_backwards.html">working backwards</a> is also useful in this mix to get outcome orientation and involve practitioners in describing outcomes (bounded by their view of what’s achievable).”</p>
<p>It’s possible that this next step is around clarifying what outcomes are most important to us (e.g. is it speed of releasing? In which case is the next priority is the slowest part of the stack to release. Or is it about cost? Or user experience? etc). Or it could be about principles. Or it could be finding a way to have the group exercise again, where we either figuratively or literally, lay cards out on the floor to identify what needs to be done to get us to our long-term goal.</p>
<p>Working this out is my next thing to do.</p>
<h2 id="in-summary">In summary</h2>
<ul>
<li>We’ve set up some structured planning meetings and agreed to make sure we are always focusing on the most important large pieces of work (the “rocks”).</li>
<li>We’ve set aside some time to discuss the medium sized things that need a bit more in-depth thinking (the “pebbles”).</li>
<li>We’ve clarified when something is work that principal engineers should do: when it is work that only we can do.</li>
<li>We’ve identified the three next big priorities, and we are using our forward planning meeting to share context of current and upcoming priorities.</li>
</ul>
<p>However:</p>
<ul>
<li>We have not worked out how to involve the wider group of senior engineers in the discussions.</li>
<li>We don’t know how to share context and get clarity over what the next priorities are either in a remote meeting or asynchronously.</li>
<li>We haven’t created a coherent plan of action so that anyone in the wider team can clearly see what the next step might be.</li>
</ul>
<h2 id="i-would-love-to-hear-your-thoughts">I would love to hear your thoughts</h2>
<p>I would love to hear from anyone who has useful contributions on this; particularly any thoughts about how to do a card exercise or similar context-sharing/prioritising activity with a large group of people either remotely or asynchronously.</p>
<p>And I would also love to hear from others who are in charge of a large stack and successfully share the work of prioritising with the team.</p>
Threat modelling cards2021-09-15T00:00:00+00:00https://www.annashipman.co.uk/jfdi/threat-modelling-cards.html<p>Threat modelling is a way to identify the potential security flaws in your system, and prioritise mitigations. On Monday I attended a session comparing some card games you can use to help you, at <a href="https://www.spaconference.org/">SPA conference</a>. This is a brief write-up.</p>
<h2 id="the-purpose-of-threat-modelling-cards-is-to-provoke-discussion">The purpose of threat modelling cards is to provoke discussion</h2>
<p>There are various ways to do threat modelling, for example the <a href="https://www.ncsc.gov.uk/collection/cyber-security-design-principles/establish-the-context-before-designing-a-system">National Cyber Security Centre (NCSC) recommends attack trees</a>.</p>
<p>Security cards are a way to help a group think laterally about the problem, and bring up ideas and discussion of potential security flaws that might not have come up on first thinking about it. They can also be a way to engage everyone in the team with the problem.</p>
<p>The idea of this conference session was to compare three such card decks to see what kinds of insights they generated.</p>
<h2 id="the-session-was-really-well-put-together">The session was really well put together</h2>
<p>The session was run by <a href="https://charlesweir.com/">Charles Weir</a>, <a href="https://www.lancaster.ac.uk/scc/about-us/people/lucy-hunt">Lucy Hunt</a> and <a href="https://www.shamalfaily.com/">Shamal Faily</a>. They had documented a basic banking application with a simple architecture diagram. We then used one of the three card decks to generate potential security issues with the application. After ten minutes we swapped to another card deck, and then the third. The three groups each used the three decks of cards in a different order.</p>
<p>At the end of the three games we spent a few minutes dot voting on the issues we’d identified – black dots for the most important, red dots for the most interesting.</p>
<h2 id="we-used-three-very-different-decks-of-cards">We used three very different decks of cards</h2>
<p>The card games we used in this session were <a href="https://www.usenix.org/conference/3gse14/summit-program/presentation/shostack">Elevation of Privilege</a> (you can download a <a href="https://adam.shostack.org/Elevation-of-Privilege-BlackHat2010ShostackFinal.pptx">very useful presentation</a> on it), <a href="https://daylight.berkeley.edu/adversary-personas/">Adversary Personas</a> and <a href="http://securitycards.cs.washington.edu/index.html">Security and Privacy Threat Discovery</a>.</p>
<p>Each of these decks were quite different with different activities involved.</p>
<h2 id="elevation-of-privilege">Elevation of Privilege</h2>
<p>The Elevation of Privilege cards are very technical. They outline something that might happen, divided into the six areas in the <a href="https://en.wikipedia.org/wiki/STRIDE_(security)">STRIDE framework</a>. For example, one card is titled “Spoofing” and reads “An attacker could squat on the random port or socket that the server normally uses”.</p>
<p>The actual game involves placing cards on the diagram to indicate where in the architecture the situation described on the card could be a problem, but for our shortened version of the game we were to indicate what situation was present for this to be a problem.</p>
<p>All the cards were more or less that level of technicality, requiring a fairly high base level of architectural knowledge and understanding of security terms. It would be hard to imagine playing this game with the product owner or even junior engineers.</p>
<p>It did however prompt some advanced thoughts and useful discussion about what security flaws there might be in our system.</p>
<h2 id="adversary-personas">Adversary Personas</h2>
<p>This was the second game our group played and was very different. It had cards about human impacts (e.g. “Societal wellbeing”), cards about Adversary Motivations (e.g. “They are incompetent”, “They need to influence politics”), cards about Adversary Resources (e.g. “They live in tomorrow’s world”).</p>
<p>You play this game by first understanding what you are protecting (the impact) and then getting into character as one of the adversaries, and then using the resources to see if that affects how you think about that persona. The end goal is to identify the top three personas who are most likely for your organisation and then identify what and how they might attack your system.</p>
<p>As for all the games, we didn’t play the full version in the session. Instead, we talked about the personas and impacts and used that discussion to identify some potential issues.</p>
<p>This one generated some really interesting discussion, and people with less technical or security experience were able to get fully involved.</p>
<h2 id="security-and-privacy-threat-discovery">Security and Privacy Threat Discovery</h2>
<p>This was the third game we played, and after the other two it was very odd. Each card in this game has A LOT of information and suggestions. For example, a card headed “Technological Attack: Adversary’s Methods” reads:</p>
<div class="quote">
<p>What kinds of technical attacks might the adversary perform over an analog or digital link? How would this enable or amplify an attack on confidentiality, integrity, or availability?</p>
<p>Example Related Concepts:</p>
<ul>
<li>Example Attacks: denial-of-service · spoofing · repudiation · elevation of privilege · replay attacks · relay attacks · jamming</li>
<li>Example Outcomes: acquire password files · eavesdrop on confidential exchanges · install bot software</li>
</ul>
</div>
<p>Coming straight after Adversary Personas, which had really encouraged unbounded thinking, it was hard to engage with these cards at first. With the above one, for example, it lists quite a lot of the potential answers, which doesn’t feel very motivating to come up with more.</p>
<p>The <a href="http://securitycards.cs.washington.edu/activities.html">activities</a> for these cards are different to the other two sets and they can also be used for education rather than for securing a particular system. It seemed like they might prompt good ideas for a group less familiar with security concepts, and we did generate quite a few potential issues from them.</p>
<h2 id="which-cards-generated-more-insights">Which cards generated more insights?</h2>
<p>When coming together and looking across the three groups at the end of the session, the interesting thing was that each deck of cards, regandless of the group or the order, had generated roughly the same number of black dots, i.e. important security issues to address.</p>
<p>However, in all groups, the Adversary Personas had generated the most red dots, i.e interesting issues raised.</p>
<h2 id="i-would-definitely-want-to-use-a-deck-of-cards">I would definitely want to use a deck of cards</h2>
<p>I found this exercise really interesting. All of the decks of cards prompted the group to think outside our default ways of thinking, so I would advocate using a deck of cards as a supplement to any threat modelling session I was doing.</p>
<p>Which deck I would want to use would depend on the context, e.g. the technical understanding of those taking part, what other methods we were using, whether the priority of the session was to identify new risks or gain a collective understanding, etc.</p>
<p>Other security cards are available, for example <a href="https://opensource.com/article/19/3/protection-poker-agile-security-game">protection poker</a>.</p>
<p>And other threat modelling methods are also available! This <a href="https://insights.sei.cmu.edu/blog/threat-modeling-12-available-methods/">a useful summary</a>.</p>
<p>Charles, Lucy and Shamal plan to write up a more detailed blog post with the outcome of the session, which I look forward to reading.</p>
<h2 id="go-to-spa">Go to SPA!</h2>
<p>And <a href="https://www.spaconference.org/">SPA conference</a> is still going on, until Friday this week, with lots of great sessions still to come. You can <a href="https://www.eventbrite.co.uk/e/spa-conference-2021-tickets-155925821329?discount=SCHOLARSHIP2021">get a free ticket here</a>.</p>
Setting up a Ubiquiti network2021-06-21T00:00:00+00:00https://www.annashipman.co.uk/jfdi/setting-up-ubiquiti.html<p>We wanted good WiFi coverage in our house and we did not have it, so we set up a Ubiquiti network. This is how we did it.</p>
<h2 id="options-for-getting-good-coverage">Options for getting good coverage</h2>
<p>There are three ways to get good WiFi coverage in your house:</p>
<ol>
<li>Use a repeater (also known as an extender or a booster), like a <a href="https://www.johnlewis.com/tp-link-re450-dual-band-wi-fi-range-extender-ac1750/p5381915">TP-link range extender</a>.</li>
<li>Use mesh networking, like a <a href="https://www.netgear.com/home/wifi/mesh/tri-band/">Netgear Orbi</a>.</li>
<li>Have multiple wired access points.</li>
</ol>
<h3 id="1-a-repeater">1. A repeater</h3>
<p>A repeater uses WiFi to present the same network in different parts of the house. It’s using the same WiFi as the other devices, so a potential downside is that they can get in the way of each other’s traffic. This is the cheapest option.</p>
<h3 id="2-mesh-networking">2. Mesh networking</h3>
<p>A mesh network creates its own network (a <a href="https://en.wikipedia.org/wiki/Backhaul_(telecommunications)">backhaul</a> network) to pass data around your house rather than using WiFi, so you do not get the network contention you might have with a repeater.</p>
<p>Having a mesh network is like having a wired network, though because it’s not using wires it’s not as fast and potentially not as reliable (e.g. radio interference can happen). This is more expensive than getting a repeater, but is very easy to set up.</p>
<h3 id="3-multiple-wired-access-points">3. Multiple wired access points</h3>
<p>A wired network is the best of all because the network can communicate at the highest speed your wires can do, which will be faster than WiFi. There is also minimal interference as the network is not shared by anything else.</p>
<p>This is the most involved to set up, and can be quite expensive.</p>
<p>We had our house renovated a year ago, and with all this in mind we had Ethernet wired throughout the house, with the aim of doing option 3.</p>
<h2 id="we-decided-to-use-ubiquti-for-the-networking-gear">We decided to use Ubiquti for the networking gear</h2>
<p>As a technologist, <a href="https://www.ui.com/">Ubiquiti</a> appealed because, if you know what you’re doing, you can optimise it.</p>
<p>Ubiquiti has the kind of crossover product that is aimed at consumers with experience (“power users”), rather than actual professional networkers, which suited me perfectly.</p>
<h2 id="the-isp-provided-router-performs-a-number-of-roles">The ISP-provided router performs a number of roles</h2>
<p>Currently our Internet Service Provider (ISP) is PlusNet, and we were just using the router they sent us.</p>
<p><img src="/img/plusnet_router.jpg" alt="The PlusNet router" /></p>
<p>A router like this is five things:</p>
<ol>
<li><strong>A modem</strong>. This converts the signal sent over the phone line back into data.</li>
<li><strong>A firewall</strong>. The router has a software firewall on it to stop malicious traffic accessing the local network, and doing something like <a href="https://en.wikipedia.org/wiki/Secure_Shell_Protocol">SSH</a>ing onto your laptop, or changing your <a href="https://en.wikipedia.org/wiki/Routing_table">routing table</a> such that their server looks like your bank.</li>
<li><strong>A router</strong>. This connects two (or more) networks together – in this case, the local network in my house with the internet (via the ISP’s network) – and knows when to direct traffic either to the right place on the local network, or to the internet.</li>
<li><strong>A switch</strong>. This connects the devices in the house together (e.g. my phone, my laptop), determines the source and destination addresses of each <a href="https://en.wikipedia.org/wiki/Network_packet">packet</a>, and forwards it only to the correct device.</li>
<li><strong>An access point</strong>. This creates a wireless local area network (WLAN) which allows authenticated devices within range to connect to the wired network.</li>
</ol>
<p>With Ubiquiti, you separate out some or all of those roles.</p>
<h2 id="you-can-mix-and-match">You can mix and match</h2>
<p>For example:</p>
<ol>
<li>You can just get some extra access points (APs) and connect them to your existing router. This will give you extra WiFi coverage around the house. You continue to use your ISP’s router for all other roles. The APs use <a href="https://en.wikipedia.org/wiki/Power_over_Ethernet">Power over Ethernet</a> (PoE) rather than having an option to plug them into the mains, so you also need some PoE adaptors.</li>
<li>You can get some APs and a Ubiquiti switch (which usually has PoE so you don’t need to buy the adaptors). Continue to use your ISP-provided router as modem, router and firewall. This means you can do some management of the switch and the network, like getting statistics on usage or booting off the phone of a cheeky builder who looked at the router card for the password. You can also plug more in; you’re not limited to the four ports your ISP’s router has spare.</li>
<li>You can go all in: use a modem from e.g. BT, and buy everything else from Ubiquiti; a switch, APs, and a <a href="https://www.ui.com/unifi-routing/usg/">security gateway</a> as firewall and router.</li>
</ol>
<h2 id="dipping-our-toe-in">Dipping our toe in</h2>
<p>We were very tempted to go all in to begin with, but decided to be sensible and start with option 1. We got two access points and two <a href="https://www.netxl.com/poe-injectors/ubiquiti-unifi-u-poe-af-802.3af-poe-injector-48v/">PoE adaptors</a>, and some network cables.</p>
<p>We got two types of AP: one <a href="https://www.netxl.com/wifi-access-points/ubiquiti-uap-ac-lite-unifi-wifi-wireless-access-point/">disc</a> and one <a href="https://www.netxl.com/wifi-access-points/ubiquiti-uap-ac-iw-indoor-outdoor-unifi-poe-wireless-access-point/">in wall</a>. The disc is cheaper but ugly and huge, so we got one of those for the loft, and an in wall one for the study.</p>
<p>You don’t buy direct from Ubiquiti, you buy from a distributor. We got ours from NetXL.</p>
<p><strong>Cost of option 1 ~£175</strong></p>
<h2 id="a-labour-intensive-digression-into-blueprints">A labour-intensive digression into blueprints</h2>
<p>Ubiquiti’s website offers you the facility to upload blueprints of your house layout and place APs to see how much coverage you’ll get/how many you need. However, you have to do a great deal of the work yourself, e.g. drawing in walls.</p>
<p>My partner whiled away a few happy hours on this, but it’s not necessary: you can figure out how many you’ll need when you know the range of the APs and how many walls they have to get through.</p>
<h2 id="creating-the-new-wifi-network">Creating the new WiFi network</h2>
<p>In installing the APs you are setting up a new WiFi network (as opposed to supplementing your existing network).</p>
<p>If you want to not have to reconnect all your devices, you set up your new network to use the same username and password as your existing network. You have to make sure you switch off WiFi on the router (usually quite straightforward via the ISP router-management software) because having two networks with the same name will confuse devices.</p>
<p>The alternative is that you set up a new network and just leave the old one hanging around but there’s no advantage to doing that.</p>
<h2 id="setting-up-the-controller-software">Setting up the controller software</h2>
<p>To configure and manage the network, you can use an iOS app, but it doesn’t allow multiple people to control it so you would need one designated sysadmin.</p>
<p>Instead, you can use Ubiquti’s free controller software, but you need somewhere to run it.</p>
<p>You can get their <a href="https://unifi-protect.ui.com/cloud-key-gen2">cloud key</a>, which also allows remote connectivity (in case you want to administer your network from elsewhere). However, they are > £100, and they plug into the router, so take up one of your Ethernet ports.</p>
<p>You could use something like a Raspberry Pi, but we decided to use our iMac because it’s always on and we both have access.</p>
<p>We are using a Vagrant box so we don’t have to install all the other dependencies (and also because I love Vagrant). We figured it out mostly using <a href="https://help.ubnt.com/hc/en-us/articles/220066768-UniFi-How-to-Install-and-Update-via-APT-on-Debian-or-Ubuntu">this guide to installing via APT</a>, but we also had to refer to the <a href="https://help.ubnt.com/hc/en-us/articles/360012282453-UniFi-How-to-Install-Upgrade-the-UniFi-Network-Controller-Software">installing manually guide</a>, e.g. for port forwarding. <a href="https://gist.github.com/annashipman/2f3a9454e0c1a41f9357a9196f34b0b0">Our Vagrantfile</a> is on GitHub.</p>
<p>Using this Vagrantfile, you then have the Unifi software running at https://localhost:8443 and you follow <a href="https://help.ui.com/hc/en-us/articles/360012282453-UniFi-How-to-Install-Upgrade-the-UniFi-Network-Controller-Software">the installing manually guide</a> to set up your network and APs. We actually just did it locally without setting up a Ubiquiti account, which is in advanced settings.</p>
<h2 id="an-ultimately-fruitless-digression-into-docker">An ultimately fruitless digression into Docker</h2>
<p>We also investigated running the controller in Docker, after reading a <a href="https://words.bombast.net/unifi-controller-lets-encrypt-and-docker">blog post about it</a>.</p>
<p>It worked, but because were running it on a Mac, it adds another layer of indirection and management over just doing it using Vagrant, because on a Mac you cannot run Docker natively, you have to run it in a VM. (If we had been using a Raspberry Pi, for example, this would not have been the case.)</p>
<p>The VM intermediary layer created a couple of problems:</p>
<ul>
<li>there were some complexities about which volumes to share between the host and the container so that we could ensure the controller config could be backed up on the Mac rather than inside the container.</li>
<li><code class="language-plaintext highlighter-rouge">docker-machine</code> is the docker-sanctioned wrapper for running containers on Mac, and this doesn’t support bridged networking directly, requiring us to jump through more hoops (<a href="https://stackoverflow.com/questions/33021581/how-to-bind-the-vm-docker-machine-creates-to-osx-ip-address">described here</a>) in order to make the controller appear to be on the same network as the devices. (This is because if the controller is on a different network, adopting devices is more complicated).</li>
</ul>
<p>Vagrant supports bridged networking, and as I have already mentioned, I love Vagrant, so we gave up on Docker.</p>
<h2 id="spam-was-coming-from-inside-the-house">Spam was coming from Inside the House!!</h2>
<p>The controller software shows network association failures, i.e. clients not successfully connecting to the WiFi. Once we were able to see these logs, we realised we were getting thousands of network association failures a day, from a single device.</p>
<p>It turned out that the TV was trying to connect but it had the wrong password. In fact we had deliberately given it the wrong password because when it updated its software this often broke features (yes, it’s a “smart” TV) so we didn’t want it on the network – and there is no way to make the TV forget a network.</p>
<p>The thousands of failures hadn’t caused any problems at all, but now we could see these errors it had to be stopped. (We created a new network, let it join that network and then turned off that network). This cleared up the error messages on the controller. It had no beneficial effect apart from that but the curse of being an engineer is having to solve painless problems!</p>
<h2 id="dipping-a-whole-foot">Dipping a whole foot…</h2>
<p>We had our Option 1 setup for around 9 months.</p>
<p>It was definitely better than it had been previously, but it wasn’t perfect and there were some areas of poor signal, particularly in the garden, which made doing my weekly remote circuit training with my colleagues quite hard.</p>
<p>We decided to get one more access point. We thought we wanted two more, one that reached the garden and one that improved coverage in the front of house, but the ISP-provided modem only had four available Ethernet ports; three APs, plus the Apple TV which is hardwired. With some shuffling though, we managed to find a configuration of the three that did cover everything.</p>
<p><strong>Additional cost £90</strong></p>
<h2 id="time-to-jump-in">Time to jump in</h2>
<p>This lasted for around six months.</p>
<p>However, we started to notice various parts of the network dropping out. On examination the relevant access point would not even appear to be on the network. Restarting the router did not help; only unplugging and replugging the Ethernet cable for the AP solved the problem. Either the router was struggling to handle four Ethernet connections or it was reaching the end of its usable life. It was time to go all in.</p>
<p>So we needed a switch. We needed one with >4 Ethernet ports to connect all our APs and the outside world, and ideally one with PoE so we could remove all the adaptors. As we had a server cabinet, we <em>obviously</em> had to get one that was rack-mountable; so the one we got had 16 ports.</p>
<p>Theoretically we could have just got a switch and used the ISP’s router as the security gateway, router and modem, but by this stage we were ready to fully commit.</p>
<p>The security gateway also acts as a router, and buying our own allows us better control of the network. The ISP’s router was a black box; we could not debug anything. It also did not properly support Universal Plug and Play (<a href="https://en.wikipedia.org/wiki/Universal_Plug_and_Play">UPnP</a>) or Multicast DNS (<a href="https://en.wikipedia.org/wiki/Multicast_DNS">mDNS</a>) which meant that the Xbox didn’t reliably connect to the network and Airplay didn’t reliably work. Also, managing your own security gateway means you can add things like <a href="https://www.opendns.com/about/innovations/dnscrypt/">DNSCrypt</a> and use <a href="https://techcrunch.com/2020/12/08/cloudflare-and-apple-design-a-new-privacy-friendly-internet-protocol/?guccounter=1">ODoH</a>.</p>
<p>We didn’t need to buy a modem as we happened to have an old BT-provided one in a drawer.</p>
<p><strong>Total cost of switch and security gateway ~£370</strong></p>
<p><img src="/img/switch_and_security_gateway.jpg" alt="The switch and the security gateway" /></p>
<p><em>Aren’t they pretty</em></p>
<h2 id="how-we-set-it-all-up">How we set it all up</h2>
<p>If you go all in to start with, buying a switch, security gateway, and a few APs all at once, you would just plug them all together, set up your controller software, and configure. Your devices would automatically connect to the new network if it had the same name as the old network.</p>
<p>If you’re adding the extra kit later, as we did, you first have to do a full factory reset on your exisiting APs, because setting up a security gateway redefines your network, so you don’t want to try to adopt it into an existing network. Once the APs are back online you configure the network using the controller software (i.e. adopting each AP as new).</p>
<h2 id="summary-six-months-on">Summary, six months on</h2>
<p>It’s great.</p>
<p>Our Airplay devices are always available. UPnP is now integrated and works consistently so the Xbox can always access the network. Connecting to other devices is instantaneous, for example SSHing to the iMac, and connecting to the WiFi is also faster, e.g. when opening a laptop. Everything is easier to debug because everything is a little Linux box. And our internet connection speed has slightly increased, from about 72mb to 75mb down and from 18mb to 21mb up.</p>
<p>And most importantly, we have what we wanted: coverage everywhere throughout the house.</p>
<p>The total cost was about £650. This compares to about £150 buying three WiFi extenders, about £350 buying mesh networking – or £0, using the ISP-provided router. So it’s quite expensive – though spreading it out over eighteen or so months made it less painful. And it has massively improved our WiFi coverage and therefore quality of life.</p>
<p>All we need is for those OpenReach people we saw outside to really have been installing fibre to the premises as the street WhatsApp rumoured, and we’ll be completely sorted!</p>
Finance for non-finance directors: Part 32021-03-12T00:00:00+00:00https://www.annashipman.co.uk/jfdi/finance-part-3.html<p>This is the third of three posts summarising some very interesting things I learned about finance on a recent <a href="https://www.iod.com/training/open-courses/finance-for-non-financial-directors">IoD course</a>. In <a href="/jfdi/finance-part-1.html">part one</a> I talked about funding and financial statements, in <a href="/jfdi/finance-part-2.html">part two</a> I talked about accountancy principles and ratios, and today I will talk about deciding which projects, and which companies, to invest in.</p>
<h2 id="weighted-average-cost-of-capital-wacc">Weighted average cost of capital (WACC)</h2>
<p>It was at this point of the course that we got into a lot of acronyms and abbreviations. If you’re less interested in the specifics of project appraisal techniques, scroll down to “Why corporate failure occurs” for how to think about the overall health of a company.</p>
<p>An important acronym for assessing which projects to invest in is the WACC – the weighted average cost of capital.</p>
<p>The point is that capital is not free. For example, debt has interest due on it. The cost of equity is harder to calculate, as technically speaking there is no obligation to pay dividends; but investors will expect a certain return on their investment otherwise they may just sell their shares, so there is a cost to the company associated with equity.</p>
<p>The WACC is calculated by working out the cost of debt and equity, the percentage of funding which is each and summing them.</p>
<p><img src="/img/wacc_calculation_example.jpg" alt="WACC calculation example" /></p>
<p>The WACC is also known as the ‘hurdle rate’, i.e. the minimum return that any project the company undertakes needs to deliver in order to be generating value rater than eroding it. By how much should a capital project or investment exceed WACC? The answer depends on perceptions of the risks involved.</p>
<p>The WACC/hurdle rate is not usually used on its own, but as part of one of the project appraisal techniques that I will talk about a bit more below.</p>
<h2 id="capm-and-beta">CAPM and beta</h2>
<p>A commonly used model to help with calculating the cost of equity is the capital asset pricing model (CAPM), which uses the term ‘beta’ to represent how responsive your business model is to changes in the overall economy. We didn’t learn how to calculate CAPM, but what we learned about beta was interesting.</p>
<p>Beta is represented as whether it is greater than, less than or equal to 1. A beta of 1 means it’s about the same risk as the overall economy. Greater than 1 means higher risk, and less than one means more stable than the overall economy.</p>
<p>For example, airlines are good examples of sectors with a beta higher than 1 – they amplify the risks in the economy. If the economy goes down, people stop taking foreign holidays. Food retail is a good example of a beta lower than 1 – if there is a recession, people do still buy food, and if there is a boom, people do not buy more food.</p>
<h2 id="risk-is-something-where-there-is-uncertainty-of-outcome">Risk is something where there is uncertainty of outcome</h2>
<p>As I mentioned in the <a href="/jfdi/finance-part-1.html">first of these finance posts</a>, risk is not always bad, it’s also an opportunity – and it’s an intrinsic part of doing business. You cannot be a board director if you avoid risk.</p>
<p>You need to know what the principal risks are to the business, and you need to have risk management systems/internal controls in place.</p>
<p>You need to measure risk appetite whenever something major happens.</p>
<p>A risk premium is an extra incentive to investors to take the risk.</p>
<h2 id="project-appraisal-techniques">Project appraisal techniques</h2>
<p>As a director you might have to make a decisions on what project to invest in. Directors need to be able to assess the anticipated returns from a proposal. There are various methods for doing this, and we talked about three:</p>
<ul>
<li>Return on capital employed</li>
<li>Payback</li>
<li>Discounted cash flow</li>
</ul>
<p>Return on capital employed asks: what is the return generated as a percentage of the capital outlay?</p>
<p>This is simple, and useful as a screening tool – if it doesn’t exceed the organisation’s hurdle rate (the WACC) it can be rejected. However, the downside is that it’s simple and doesn’t take into considerations cash flow, timing or risk.</p>
<p>Payback asks: how long does it take for the capital outlay to be repaid from the returns generated?</p>
<p>This is also simple, and if the availability of funding or timescale is an issue, it’s very useful. The disadvantages are also with the simplicity – it doesn’t consider total returns over the lifetime of the project, or risk.</p>
<h2 id="neither-of-these-take-into-account-the-time-value-of-money">Neither of these take into account the ‘time value of money’</h2>
<p>Cash today is worth more than cash tomorrow. For example:</p>
<ul>
<li>opportunity cost (if you get the money now you can invest it and it could be worth more in the future, whereas if you wait this opportunity is lost)</li>
<li>inflation</li>
<li>risk (a bird in the hand…)</li>
</ul>
<p>So money has a value that varies depending when the money is available – the time value of money.</p>
<h2 id="discounted-cash-flow-is-an-appraisal-technique-that-takes-into-account-the-time-value-of-money">Discounted cash flow is an appraisal technique that takes into account the time value of money</h2>
<p>The first thing is to attempt to calculate the value of all the cash flows of the project as a cash amount today – its present value. Then you subtract the outgoing costs from the incoming money and that gives you the net present value (NPV).</p>
<p>There is a calculation to work out the present value using the forecasted cash flow, the ‘discount factor’ (this might be the WACC/hurdle rate), and time. The ‘discount factor’ is based on lots of assumptions and is an estimate, as is the forecasted cash flow; so there are a lot of guesses in working out the discounted cash flow.</p>
<p>Once you’ve made all these estimates and done the calculation, you have the net present value, and if it is positive, the model predicts the investment will generate value.</p>
<p>It is very sensitive and complex, but this <em>does not mean it is more accurate</em>.</p>
<p>It’s worth asking what the impact of changing a variable by a very small amount is, e.g. 25% to 24.5%. This could make the difference between the net present value being positive or negative and therefore whether it looks like the investment will generate or destroy value.</p>
<h2 id="using-internal-rate-of-return-irr">Using internal rate of return (IRR)</h2>
<p>OK, so here’s where we really get stuck into the acronyms and abbreviations. I hope you’ve been paying attention (all are defined above)…</p>
<p>IRR is an application of the NPV calculation that gives an insight into the profitability of the project itself. The IRR of a project is calculated by identifying the discount factor that causes the NPV of the project’s cash flows to equal zero.</p>
<p>You can then ask if the IRR exceeds the WACC.</p>
<p>It also allows projects with different initial investments to be ranked against each other.</p>
<h2 id="the-key-function-of-a-budget-is-to-assess-profitability-and-financial-viability">The key function of a budget is to assess profitability and financial viability</h2>
<p>As well as talking about evaluating projects, we also talked about the purpose of having a budget at all.</p>
<p>A budget includes:</p>
<ul>
<li>Forecasting</li>
<li>Planning (control/coordination)</li>
<li>Communication</li>
<li>Motivation</li>
<li>Evaluation</li>
<li>Assessing and managing risk</li>
</ul>
<h2 id="why-corporate-failure-occurs">Why corporate failure occurs</h2>
<p>We then talked about how to evaluate companies, e.g. for investment, but also in order to understand the workings of your own company.</p>
<p>Corporate failure happens when an organisation is no longer able to continue business due to continued losses or lack of liquidity to honour payments as they fall due – or both.</p>
<p>Reasons can include:</p>
<ul>
<li>Overtrading</li>
<li>Fraud</li>
<li>Poor controls</li>
<li>Lack of liquidity</li>
<li>Unmanageable gearing</li>
</ul>
<p>If you don’t pay in agreed terms of trade, creditors can wind you up, i.e. they can initiate administration. 70-80% of administrations are initiated by creditors, rather than banks.</p>
<h2 id="company-reports-are-important-for-understanding-companies-health">Company reports are important for understanding companies’ health</h2>
<p>Limited companies report annually to Companies House, and anyone can <a href="https://find-and-update.company-information.service.gov.uk/">read these reports online</a>, including creditors and potential investors. Accounts, notes to the accounts, and commentary are collectively called the company report and accounts.</p>
<p>The commentary accompanying the accounts should not just be about numbers, they should be about the why, the narrative. What did we hope would happen? What actually happened? So what?</p>
<p>It should cover leading indicators, i.e. do not wait until KPI (key performance indicators) or targets are met to report, but report on how they are going. KPIs themselves should mostly be non-financial, e.g. Customer Satisfaction. And they should be aligned to the strategy – there should be a “golden thread”.</p>
<p>KPIs should show what you are really going for – what will change when that KPI is met. And the narrative should explain why these results are being measured and what value do they bring to decision-making.</p>
<div class="quote">
<p>"We don't just want to measure our results, we want to measure what drives our results."</p>
<p>- Howard Atkins, Former CFO, Wells Fargo</p>
</div>
<h2 id="the-working-capital-cycle-helps-you-understand-where-there-may-be-gaps-in-funding">The working capital cycle helps you understand where there may be gaps in funding</h2>
<p>Working capital is the capital required to finance the operating cycle of the busness, i.e. pay debts, costs, etc. The working capital cycle is the length of time it takes to convert the working capital into cash.</p>
<p><img src="/img/working_capital_cycle.jpg" alt="The working capital cycle" /></p>
<p>Understanding this helps you manage your cash flow, inventory, and efficiency, or alternatively, see how well a business you are looking at is managing those things.</p>
<p>How long the cycle takes varies. For example a sandwich takes minutes. Building a ship takes years. Some companies have a negative working capital cycle, for example, you might buy a loaf of bread from the supermarket before they’ve paid the supplier.</p>
<p>The ‘funding gap’ is where there is a difference between collection of payment from your customers and payment of your suppliers. Not just in days, but in terms of money. There are things you can do: move stock faster; reduce payment terms and get paid quicker; delay supplier payment, or fund the gap (i.e. invest money in it).</p>
<p>In a growing business you can expect working capital to be increasing.</p>
<h2 id="break-even-analysis">Break-even analysis</h2>
<p>A break-even analysis is a really useful tool for understanding how many units the company needs to sell in order to be profitable.</p>
<p>Fixed costs: e.g. salary, rent. These remain constant regardless of activity.</p>
<p>Variable costs: e.g. materials, overtime, sales commission. These vary with activity.</p>
<p><img src="/img/break_even_calculations.jpg" alt="Break-even calculations" /></p>
<p>“Contribution” indicates that every time you sell something, it contributes to fixed cost.</p>
<p><img src="/img/margin_of_safety.jpg" alt="Margin of safety calculation" /></p>
<p>The margin of safety indicates how much sales have to drop before you are no longer breaking even. 10% seems high risk. 30% a bit more comfortable.</p>
<p>Fixed costs are useful when thinking about whether you can afford to hire more.</p>
<p>Low operational gearing means low fixed costs and often high variable costs.</p>
<p>High operational gearing means a high level of fixed costs. This is more risky because if there is a downturn, you can’t stop fixed costs.</p>
<h2 id="there-are-lots-of-ways-of-valuing-companies">There are lots of ways of valuing companies</h2>
<p>For example:</p>
<ul>
<li>Recent similar deals</li>
<li>Net book value</li>
<li>Market value/enterprise value</li>
<li>Earnings multiple/ P/E ratio</li>
<li>EBITDA</li>
<li>Free cash flow</li>
</ul>
<p>None of these are perfect. Share price is just a finger in the air.</p>
<p>The method you choose depends in part on why you are valuing the company. For example, is it deciding whether to invest, or deciding whether to buy the whole enterprise? (In the latter case you have to pay for the shares <em>and</em> pay the company’s debt.)</p>
<p>Is the company balance-sheet driven or P&L driven? Most companies are the latter, but those such as a bank, which uses its capital (i.e. what is on the balance sheet) and lends it out, are termed balance-sheet driven.</p>
<p>Price is what you pay, value is what you get. And valuing is subjective; use the tools with caution and take into account non-financial things (e.g. management credibility).</p>
<p>At this point in the course the instructor brought up a quote from the FT, which she swore was always in the course and wasn’t just because of my presence!</p>
<p><img src="/img/FT_quote_numbers.jpg" alt="Numbers can create a sense of objectivity where there is none..." /></p>
<h2 id="this-year-purpose-is-what-investors-want-to-talk-about">This year, purpose is what investors want to talk about</h2>
<p>Companies with a clear purpose that everyone understands are more resilient. The purpose leads to the business model, which leads to strategy and objectives and from there what the principal risks are and what we should be measuring.</p>
<p>Our course instructor recommended <a href="https://uk.bookshop.org/books/the-infinite-game/9780241385630">The Infinite Game</a> which talks about “the just cause”, i.e. the higher purpose of the company.</p>
<h2 id="the-course-was-incredibly-useful-and-i-highly-recommend-it">The course was incredibly useful and I highly recommend it</h2>
<p>I learned so much more on the course than I’ve been able to write up here, but I hope this has given you some useful information.</p>
<p>The main things I took from it were that finance is not cold hard facts, but “it depends”. There are lots of estimates and assumptions, not least in profit, which is <em>not cash</em>. The important question to ask about any financial figure or statement is “so what?”</p>
<p>I’ve learned a lot more about how to evaluate a company and how to read and understand financial statements.</p>
<p>The next piece of the puzzle is tying all this back to the work I actually do; I am a technical director: I and my team are not producing widgets that we can feed into a break-even analysis; we need to understand how our work contributes to the financial health of the company and – possibly more importantly – what work we should not do, because it just does not offer value for money. This course has really helped me understand the structure of what I need to fit my work into; the next piece of the puzzle will follow in due course.</p>
<p>In the meantime, I can now understand the financial pages of the FT!</p>
<p><em>This is post three of three. You can read <a href="/jfdi/finance-part-1.html">part one here</a> and <a href="/jfdi/finance-part-2.html">part two here</a>.</em></p>
Finance for non-finance directors: Part 22021-03-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/finance-part-2.html<p>On Monday I published <a href="/jfdi/finance-part-1.html">part one of three blog posts about finance</a>, finishing with a quiz. In this post I will share the answers to the quiz, then talk about some important accountancy principles and some ways of thinking about the financial health of companies. In <a href="/jfdi/finance-part-3.html">part three on Friday</a>, I will talk about methods for evaluating projects and companies.</p>
<h2 id="answers-to-the-balance-sheet-quiz">Answers to the balance sheet quiz</h2>
<p>Here are the answers and how we worked them out (there are other ways to work them out, this is just how we got there).</p>
<p>Firstly, we looked at fixed assets and stock. We thought about who might have a large proportion of those and came to the conclusion that Morrisons must be 2, with a lot of physical shops and the largest proportion of inventory. EasyJet must be 5, as they own planes, so 63% seemed about right for fixed assets, plus people pay cash for tickets in advance, so 19% in cash sounds reasonable.</p>
<p>Finally, Apple have some shops and some inventory, so we figured Apple for 1. Our course instructor noted that their balance sheet looks more like one of an investment company. 1 also has the highest level of long-term debt, which matches borrowing to pay a dividend.</p>
<p>Barclays also have a lot of branches, but they are old buildings, so the net book value (the value of a fixed asset after depreciation) is low. It borrows and lends, which correlates with high liabilities and receivables, so that makes 3 look most likely.</p>
<p>Facebook had to be number 6 because of the huge proportion of intangible assets, e.g. acquisition of WhatsApp, and the huge percentage of equity, given that they’ve never paid a dividend.</p>
<p>We thus reached 4 as the IoD by a process of elimination, but the business model of the IoD includes delivering courses (like this one!) which are paid for upfront which matches the high proportion of cash, as well as creditors due in < 1 year.</p>
<ol>
<li>Apple</li>
<li>Morrisons supermarket</li>
<li>Barclays</li>
<li>The IoD (Institute of Directors)</li>
<li>easyJet</li>
<li>Facebook</li>
</ol>
<h2 id="accountancy-guidelines">Accountancy guidelines</h2>
<p>It is mandatory to adopt accountancy guidelines (of which there are a few – IFRS: International Financial Reporting Standards; UK GAAP – UK Generally Accepted Accounting <em>Practice</em>; US GAAP: Generally Accepted Accounting <em>Principles</em>).</p>
<p>However, they are guidelines or principles, not rules. Your accounts do not need to be compliant with every one. If adopting an accounting standard would lead to an outcome that was not true or fair, then you must not follow it. In that case, you would talk to the auditors, who would add a note to this effect.</p>
<p>An important principle is substance over form. Accountants are not lawyers, and the important thing is to communicate fairly and truthfully the state of the company’s finances.</p>
<p>The US guidelines are much more hard and fast, and people try to avoid them; some US companies even base themselves in the UK so as not to have to comply with them.</p>
<p>Following are some accountancy principles and what they mean.</p>
<h2 id="liquidity-is-the-ability-to-pay-your-debts-as-they-fall-due">Liquidity is the ability to pay your debts as they fall due</h2>
<p>How do you know if there is a problem with liquidity? <em>it depends</em>. What do I owe, when? What am I owed? When will I get it? Will I get it all?</p>
<p>Remember, <em>profit is not cash</em>. Many businesses that make a profit do not have sufficient cash to pay their debts.</p>
<h2 id="double-entry-bookkeeping">Double-entry bookkeeping</h2>
<p>This is the commonly used system of accounting where every entry has a corresponding and opposite entry, thereby identifying the two aspects of any financial transaction or adjustment. For example, one entry might show a source of funding (e.g. a sale) and the corresponding entry is the result (e.g. a debtor). When the debtor pays up, that entry would be cash one side and clearing of the debt on the other.</p>
<p>Both sides of each transaction have to add up to the same total. Double-entry bookkeeping is about properly preparing the accounts.</p>
<h2 id="how-you-pay-for-things-is-important">How you pay for things is important</h2>
<p>What makes a strategy viable is how you are going to pay for it. Whether you pay now or over time can move something from being viable to non-viable.</p>
<p>Our instructor gave us a case study of a new company that went under because they made a few wrong decisions. 1. They invested some cash when they should have borrowed it, repaying over 10 years, i.e. they should have increased their gearing. And 2. They paid for the first lot of materials in the first month, instead of negotiating 30 days with the supplier. She described this as ‘mismanagement of funding’ and even though they made a profit, the company went under.</p>
<p>Note that in the public sector, the wording is different but the fundamental concepts are the same – you have resources, you need to do the best you can with those resources.</p>
<h2 id="a-going-concern-is-a-company-that-will-continue-trading-for-12-months-from-its-annual-report">A going concern is a company that will continue trading for 12 months from its annual report</h2>
<p>That is, one which has no <em>necessity</em> or <em>intent</em> to cease trading. Necessity could be lack of liquidity, i.e. they will not be able to fulfill their debts when they fall due.</p>
<p>This applies not just at annual report time. If you suspect the company that you are on the board of is not a going concern at any time, you have an obligation to report it – to the board in the first instance. Get it minuted, and get independent legal advice. Knowingly trading while insolvent (unable to meet debts) can lead to directors becoming personally responsible for the company’s debts.</p>
<p>Also note that a lot of things on the balance sheet are valued as if the company is a going concern, e.g. the value of the stock. If you <em>had</em> to sell it off today, it would not fetch as much.</p>
<h2 id="accruals-and-matching-is-about-when-activities-are-recognised">Accruals and matching is about when activities are recognised</h2>
<p>I talked briefly about this in <a href="/jfdi/finance-part-1.html">the first post</a>. Accruing means we record things when they happen, not when the cash flow happens.</p>
<p>For example, if your financial year is Jan–Dec and you have an electricity bill due next January. By the end of the accounting year you’ve neither received a bill nor paid it, but you have made an accrual to show that the money will be due. You’ve had use of the electricity, you just have not yet paid for it.</p>
<p>Accruals are for both expenses, like the electricity bill, and income, for example if you are due but have not yet received interest, you put the interest in as an an accrual.</p>
<p>Because of accruals, it doesn’t matter to the P&L whether you are paid in advance or after delivery for sales, it will look the same in each case. (However, paid in advance is better – as you will actually have the cash, and therefore will be more likely to survive the year.)</p>
<p>The matching concept only exists in accruals accounting (the alternative to accruals accounting is cash-basis accounting). Matching means you match revenues with the expenses incurred to earn those revenues, and that you report them both at the same time.</p>
<h2 id="the-realisation-principle">The realisation principle</h2>
<p>Also called recognition, this is that revenue can only be recognised once the underlying goods or services associated with the revenue have been delivered or rendered.</p>
<p>I understood this as “don’t count your chickens before they are hatched”.</p>
<p>We never realise a profit until it has been earned.</p>
<p>However, we might realise bad news in advance if it is likely to have an impact.</p>
<h2 id="the-reliability-principle">The reliability principle</h2>
<p>Requires that there are internal controls for backing up all of this.</p>
<h2 id="the-prudence-principle-is-about-making-provisions">The prudence principle is about making provisions</h2>
<p>For example, you might anticipate a significant loss this year, so you charge the P&L, i.e. make a provision. But then, if the loss doesn’t end up happening, in the following year you may ‘release the provisions’. Last year you took a hit to the P&L, but the cash appeared this year – and this year’s P&L goes up.</p>
<p>Our instructor pointed out this is a standard move for a new CEO or Finance Director, as you’ve got a window of opportunity to blame the whole thing on your predecessor. You make provisions in year 1, stating because of everything that was going wrong before you joined, this year you will take a hit on profit. Then in year 2 you release the provisions and it looks great. In fact, the profits may be the same as normal in both years.</p>
<p><img src="/img/releasing_provisions.jpg" alt="Graph showing two potential profit lines" /></p>
<p>In this graph, line 1 and line 2 may actually be the same profit over two years.</p>
<p>You can also make some big write-offs in your first year.</p>
<p><em>Timing is everything</em>.</p>
<h2 id="most-accounting-systems-are-poor-at-capturing-costs">Most accounting systems are poor at capturing costs</h2>
<p>Every time you interact with a customer it costs money. You might find that customers who cause a lot of trouble are actually causing a loss.</p>
<p>There are rules of thumb; for example it might be that 50% of your customers are causing you to break even, a small number are making you a profit and some are causing a loss. You could find that you increase your profits by reducing your sales and being more selective about your customers.</p>
<h2 id="are-your-costs-delivering-value">Are your costs delivering value?</h2>
<p>Expenses describe something that is giving value – they are not just costs.</p>
<p>So for example, a new CEO might announce they are striking 10% off the cost base. But if it’s not strategic this could very well be a bad move – those costs may be delivering value.</p>
<h2 id="the-role-of-auditors-is-to-say-whether-the-accounts-are-true-and-fair">The role of auditors is to say whether the accounts are true and fair</h2>
<p>Auditors are not there to say whether the information presented is “correct” or “accurate”, but whether it is “true and fair”.</p>
<p>It’s an independent examination of the accounts and expression of opinion. Are the accounts properly prepared? Are they free from material misstatements?</p>
<p>‘Material’ means relevant, pertinent. i.e. would a difference here change your decision?</p>
<h2 id="financial-ratios-can-help-with-assessing-the-health-of-a-business">Financial ratios can help with assessing the health of a business</h2>
<p>When looking at the financial statements, the context is important. Is this an early-stage or a mature business? What are you expecting to see and why? What is the direction of travel over the next 5-10 years?</p>
<p>Another useful tool is the use of financial ratios. Financial ratios are the relationship between two numbers. They don’t give answers, but might highlight areas to look at.</p>
<p>For example, I talked about financial gearing in <a href="/jfdi/finance-part-1.html">Monday’s post</a>. The way you calculate the ratio is long-term debt divided by total capital employed (i.e. long-term debt plus equity), then multiply by 100 to give a percentage.</p>
<p><img src="/img/gearing.jpg" alt="Gearing % = debt/total capital employed x 100" />
Being highly geared is neither good nor bad; <em>it depends</em> on the circumstances; but it’s a number you can look at to see whether it’s as you would expect for a company in that situation.</p>
<p>Other ratios include things like Earnings per Share (EPS), and the Acid-test ratio. <a href="https://www.investopedia.com/financial-edge/0910/6-basic-financial-ratios-and-what-they-tell-you.aspx">Here is a list of some common ratios</a> and <a href="https://corporatefinanceinstitute.com/resources/knowledge/finance/financial-ratios/">here is a list of ratios and their calculations</a>.</p>
<p>It’s worth knowing that there is not only one way of categorising ratios. So for example in my exam for this course, I was told to use the IoD version. As with everything else in accounting the important thing is to be consistent.</p>
<h2 id="gross-profit-margin">Gross profit margin</h2>
<p>An interesting ratio to look at is gross profit margin, which is gross profit / net sales. A reminder that profit is an estimate; but even given that we can talk about high margin and low margin businesses.</p>
<p>For example, Waitrose is high margin and Aldi is low margin. Waitrose stores are small. Aldi stores need to be large because if it’s low margin you need to sell more.</p>
<p>Another way of looking at it is that M&S food is mostly perishable, so you can’t stack up lots of it, therefore it needs to be high margin. The percentage of food that is perishable in Aldi is much lower.</p>
<p>If you do different product lines, do not just look at gross profit margin, as profits may be down on some and up on others but the overall line might be fine.</p>
<h2 id="ratios-when-thinking-about-whether-to-invest-in-a-company">Ratios when thinking about whether to invest in a company</h2>
<p>Two of the most commonly used ratios for thinking about investment are earnings per share, and from that, price/earnings ratio (P/E ratio), based on the assumption that the share price represents the discounted value of the future earnings.</p>
<p>You calculate earnings per share by taking profit after tax and dividing by the number of shares. Then the P/E ratio is market price per share over earnings per share. If the P/E for example is 10, then you might say, “the company is trading on a multiple of 10”.</p>
<p><img src="/img/pe_calculation.jpg" alt="earnings per share and P/E ratio calculations" /></p>
<p>The main reason for trading higher is an expectation that the company might grow in the future (although it could also indicate something like rumours of a takeover).</p>
<h2 id="evaluating-projects-and-companies-will-follow-on-friday">Evaluating projects and companies will follow on Friday</h2>
<p>In this post I’ve talked about some important accounting principles, and some ways to think about the financial health of a business. In the final post on Friday, I will talk about methods for evaluating projects and companies.</p>
<p><em>This is post two of three posts. You can read <a href="/jfdi/finance-part-1.html">part one here</a> and <a href="/jfdi/finance-part-3.html">part three here</a>.</em></p>
Finance for non-finance directors: Part 12021-03-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/finance-part-1.html<p>A couple of years ago I got some <a href="/jfdi/finance-for-non-accountants.html">excellent training on finance from a finance director at the FT</a>. At the end of last year I went on a longer, more intensive <a href="https://www.iod.com/training/open-courses/finance-for-non-financial-directors">course from the IoD</a>, aimed at non-finance directors, and I learned LOADS.</p>
<p>In this post I will cover some important principles about company finance, sources of company funding and financial statements, and include a quiz!</p>
<p>In the <a href="/jfdi/finance-part-2.html">follow-up post on Wednesday</a>, I’ll give the answers and cover some accountancy principles and <a href="/jfdi/finance-part-3.html">on Friday</a> I’ll cover how to evaluate whether to invest in a company or a project.</p>
<h2 id="the-role-of-finance-is-to-enable-the-organisation-to-meet-its-strategy">The role of finance is to enable the organisation to meet its strategy</h2>
<div class="quote">
<p>"Finance without strategy is just numbers;<br />
Strategy without finance is just dreaming"<br /></p>
<p>– Emmanuel Faber, CEO at Danone</p>
</div>
<p>The Finance Director’s role on a board is to provide financial and strategic guidance and make sure the right policies and processes are in place for sound financial management. Non-finance directors also have a duty of care, and need to be able to understand finance sufficiently in order to be able to challenge effectively, as well as think about the value of the work they do or sponsor.</p>
<p>Financial strategy is about the use and management of resources. It is driven by the corporate strategy.</p>
<p>While we learned a lot of practical information, most important were three key messages the course leader impressed upon us throughout the course: “so what?”, “it depends” and “profit is not cash”.</p>
<h2 id="so-what">“So what?”</h2>
<p>There is a difference between financial direction and accounting. The question we always need to be asking about financial facts and figures is “so what?”. What’s the <em>story</em> about the numbers, what are the numbers telling us? “Why are you telling me this?” “What are 3 key things I need to know about this?”</p>
<p>Accountants give data, finance directors give information. You are right to keep asking until you understand the message being conveyed by the figures.</p>
<p>Board members are jointly responsible for the financial health of the organisation, and even for those not yet at that level it’s useful to remember the importance of making sure you understand the details, <a href="/jfdi/roof-bug-fixing.html">even if you think it’s not your area of expertise</a>.</p>
<p>Board members may not have to do all the work involved, but they remain responsible: “You can’t delegate the responsibility but you can delegate the task”.</p>
<p>The board should be challenging, asking questions. Our instructor said that as an outside advisor or potential investor, it is a concern when she finds a board where they all agree with each other. Even as a non-executive director it’s not good enough to say you didn’t understand.</p>
<h2 id="it-depends">“It depends”</h2>
<p>It turns out that, like my disappointment when I found out that maths was not in fact a field of science in which it was possible to know everything, the same is true of finance. The figures given are not true facts. They are subjective estimates, based on what you think will happen next, and as a consumer of those figures you need to understand the assumptions being made.</p>
<p>For example, if someone buys something from you, you make an estimate about when they’ll pay you, and whether they will indeed pay you the full amount – you may need some provision for bad debts. I will talk more about this, but the main take-home message around this was: whatever the question about finance, the answer is very often “it depends”.</p>
<h2 id="profit-is-not-cash">“Profit is not cash”</h2>
<p>This was the main message that the director of the course wanted us to take away. Profit is not cash. Making a profit does not necessarily mean you are making money, and profit alone is not enough, you also need cash.</p>
<p>Profit is where you sell something for more than you estimate it costs. This is an example of an assumption as I mentioned above. You do not actually know exactly how much it cost. To give a simplistic example, you get a block of wood and reckon you can make four tables. You then base the sale price and estimate of profit on that. But actually, something is wrong with one part of the wood and you only managed to make three tables.</p>
<p>In fact, profit, rather than being something cut and dried as I had previously thought, is one of the things that includes the most estimates and assumptions.</p>
<p>In addition, profit is what you record in the profit and loss statement, not cash flow. Cash flow is NOT a determinant on whether you have earned the sale. It’s irrelevant whether someone pays or not – it’s recorded as revenue in the profit & loss statement. Even if they don’t pay, you still made the sale – all it means is you have a bad debt. This is better information as well, so you know at the end of the year approximately how many of your sales didn’t pay you.</p>
<p>In fact, when looking at the previous year, our instructor said 65% of the companies that had failed in the UK had <em>increased</em> their profit that year. The commonest cause of failure due to liquidity is growth – the company grows too quickly and can’t fund it.</p>
<h2 id="limited-company-vs-partnership">Limited company vs partnership</h2>
<p>We then talked about company structure.</p>
<p>“Limited” company means there is limited liability to the shareholders (the owners). In a partnership, liability is unlimited.</p>
<p>So why would you form a partnership? It used to be mandatory to allow for professional advice, and there used to be a tax advantage. Now the main benefit is that you do not have to make any information public or even register with Companies House (or pay the associated fees).</p>
<p>As an investor, it’s not a good idea to invest in a partnership unless you are going to be involved day-to-day because of the unlimited liability.</p>
<p>With a limited company, the company is a separate legal entity, so you can separate ownership and management. Management (i.e. the board) are the stewards to safeguard the assets on behalf of the shareholders. If you are both (e.g. on the board, and also a shareholder) be careful – you have two different sets of responsibilities and risks.</p>
<h2 id="private-vs-public">Private vs public</h2>
<p>A private company is one that cannot offer its shares to the general public for trading. They have Ltd (or Limited) after their name.</p>
<p>A public company is one where shares can be marketed or traded. They are more likely to have external shareholders. A public company has Plc (or Public Limited Company) after their name.</p>
<p>Public doesn’t mean listed. A listed company is one where shares are officially listed and traded on a stock market, for example the London Stock Exchange.</p>
<p>There are about 4m registered limited companies in the UK; <a href="https://www.gov.uk/government/news/uk-company-statistics-2018-to-2019">fewer than 4% of them are public</a>, and about 800 are listed on the main market of the London Stock Exchange.</p>
<p>There are fewer listed companies than there used to be as there is more regulation now, meaning some companies de-listed rather than comply. For example <a href="https://en.wikipedia.org/wiki/Insider_trading">insider trading</a> wasn’t <a href="https://www.legislation.gov.uk/ukpga/1980/22/pdfs/ukpga_19800022_en.pdf">illegal in the UK until 1980</a>, and since private companies can’t trade publicly, it’s easier not to fall foul of that law.</p>
<p>Listing a company allows it to raise capital much more easily. For example, Facebook listed to create an exit route for the original investors. It’s a way of making it clear what the value of the shares is, and making it much easier to sell them.</p>
<h2 id="company-funding-debt-or-equity">Company funding: debt or equity</h2>
<p>There are two sources of equity: share capital – money you get from selling shares to investors, and retained profits – profits you made in previous years that you did not return to investors as dividends.</p>
<p>Note that retained profits – also called retained earnings and sometimes reserves, though <a href="https://keydifferences.com/difference-between-retained-earnings-and-reserves.html">reserves is not precisely the same</a> – is not a sum of money; it’s not cash, it’s just the cumulative amount of profit from previous years that has not been paid out as a dividend.</p>
<p>With equity, you have no legal obligation to pay anything back to shareholders.</p>
<p>Debt is money you borrow, for example from the bank. With debt you do have a legal obligation to repay it, and also interest has to be paid. However, it can be a much easier and quicker way to raise money than a funding round from investors.</p>
<p>The legal obligation to repay makes debt a liability. Equity is not a liability.</p>
<h2 id="the-financing-decision-depends-on-various-factors">The financing decision depends on various factors</h2>
<p>What balance you choose between debt and equity depends on:</p>
<ul>
<li>the life‐stage of the business</li>
<li>what the funding is for</li>
<li>whether the company is private or public</li>
<li>whether the funding is for long‐ or short‐term use</li>
</ul>
<p>Considerations include:</p>
<ul>
<li>risk – there is more risk to the company in debt than in equity</li>
<li>control – with equity, shareholders might take control</li>
<li>flexibility – debt is more flexible, you can pay it back if you don’t need it, and it’s quicker to raise</li>
<li>tax – interest can be a tax-deductible expense, whereas dividends are not</li>
<li>possibility of raising more – if equity, shareholders should be open to investing more, and a successful rights issue can raise the value of the shares because it indicates strong belief in the future growth of the company</li>
</ul>
<h2 id="financial-gearing-and-leveraging">Financial gearing and leveraging</h2>
<p>Financial gearing is a ratio of what proportion of your capital is equity and what is debt.</p>
<p>A company that has more debt than equity is said to be “highly geared”, or “highly leveraged”. It’s not necessarily a good or bad thing (<em>“it depends”</em>). For example, not borrowing is not a good decision if it means you miss opportunities. Your job as a director is to get the balance right.</p>
<h2 id="understanding-financial-statements">Understanding financial statements</h2>
<p>Financial statements are information so that the board and investors can make decisions. It is about the narrative as much as the statements themselves – the course leader said when you are learning about a company, don’t pick up the financial statements on the first day. If you’re there for a week, maybe spend one afternoon on them. The narrative is what is important.</p>
<p>The kind of information you are looking for from the narrative and financial statements are where does the finance come from? How does the company generate income? What are the costs? Which of those costs are discretionary/non-discretionary?</p>
<p>You should look at the three main financial statements: balance sheet, P&L and cash-flow.</p>
<p>The cash-flow statement from last year is probably the most meaningful. How quickly are they paying and getting paid, and what is their liquidity position? Are they a going concern? (I will talk more about liquidity and being a going concern in the next post).</p>
<h2 id="the-balance-sheet-is-a-snapshot-of-the-business-and-its-financial-health">The balance sheet is a snapshot of the business and its financial health</h2>
<p>In the <a href="/jfdi/finance-for-non-accountants.html">previous course I did on finance</a>, we did not talk much about the balance sheet, so it was really interesting in this course to learn more about it.</p>
<p>The balance sheet is like an x-ray – a one-off snapshot – and your expectations of what good looks like vary, for example with the age of the patient. Similiarly with a balance sheet, is it a mature business or an early-stage start up? You’d expect to see different things on the balance sheet.</p>
<p><img src="/img/growth_over_time.png" alt="Graph showing growth against time, A and B marked" /></p>
<p>For example, this graph shows how you would expect growth to change depending on the age of the company. At A the company has higher debt. Growth requires funding. Plus it has had less time to build up from previous years. Growth can really increase from A.</p>
<p>At B, the company probably already has the market share and growth may well have peaked.</p>
<h2 id="balancing-the-balance-sheet">Balancing the balance sheet</h2>
<p>On a balance sheet, roughly speaking, there are two sides. On one side you have the ‘uses’, i.e. what your money is being spent on, and on the other side you have the ‘sources’, i.e. where the money came from, which is split into equity and liabilities (debt).</p>
<p>Both sides of the balance sheet should add up to the same number (hence the name…). Neither debts nor shareholders’ cash is yours, but assets are.</p>
<h2 id="the-uses-side-on-the-balance-sheet-is-assets">The ‘uses’ side on the balance sheet is assets</h2>
<p>Uses are ‘fixed assets’, or ‘current assets’.</p>
<p>Assets are things that are expected to generate value. For example, with a shop, you don’t expect to have to sell the property to recover the value; you expect to <em>use</em> the property as a place where people can buy from you and it generates value in this way.</p>
<p>‘Fixed assets’ are assets you generate value from over a period of time longer than the accounting period. They can be tangible, intangible or investments.</p>
<p>Examples of tangible assets are the building the company is headquartered in, the desks and chairs in the building, etc. Tangible confusingly can also be called PPE which doesn’t mean what we’ve come to understand in 2020, it means “Property, Plant and Equipment”.</p>
<p>Examples of intangible assets might include the website you work on and the related software, and examples of investments include things like investing in a supplier to ensure deliveries, or investing in a customer to ensure continuity of demand – strategic investments.</p>
<p>An interesting thing about intangible assets is that they only count if you bought them. So for example, M&S’s brand is not on their balance sheet because they built it up, they did not purchase it. There is currently a consultation about this kind of thing, as intangible assets are the vast part of the value of many companies (consider, for example, Facebook’s website, which they did not purchase and is therefore not valued as an asset).</p>
<p>This also makes it hard to reason financially about brand deterioration or reputational damage, unless it has a direct impact on sales.</p>
<p>‘Current assets’ are things where the value is generated from providing an immediate cash flow in the current account period, like receivables (i.e. money you are owed from debtors), cash, or stock (inventory).</p>
<h2 id="the-sources-side-is-liabilities-and-equity">The ‘sources’ side is liabilities and equity</h2>
<p>‘Sources’ is split into equity (shareholders’ funds, retained profit), long-term liabilities (debt) and short-term liabilities (i.e. due within the next 12 months; this might include debt coming due, or things like electricity bills).</p>
<h2 id="valuation-of-all-these-things-can-be-quite-difficult">Valuation of all these things can be quite difficult</h2>
<p>It is another thing that is not set in stone. For example, valuing stock involves being confident of the price you can sell it at, and you must value it at the lower of the cost or what you can get for it. So if you bought it for £1 per unit but are pretty sure you can only sell it at 80p per unit, the value is 80p per unit – but note that it’s still an estimate of what the value is.</p>
<p>Likewise when valuing receivables – it is not how much you are owed, but how much you can reasonably expect to actually receive; as some debtors will not pay, or will not pay the full amount owed.</p>
<h2 id="the-profit-and-loss-statement-pl-is-about-the-profit-and-loss-over-a-specific-accounting-period">The profit and loss statement (P&L) is about the profit and loss over a specific accounting period</h2>
<p>I wrote about the P&L in <a href="/jfdi/finance-for-non-accountants.html">my previous blog post about finance</a> but without the context I have now.</p>
<p>While the balance sheet is a snapshot, the P&L covers the current accounting period.</p>
<p>But things don’t conveniently fall into accounting periods, e.g. you make a sale in 2020 but don’t get paid until 2021, or you pay staff to produce something in 2020 which won’t even be on sale until 2021. This is where accruals and matching come in. Accruals and matching dictate that sales and expenses are not recorded when the cash flow happens but in the accounting period in which they occur or to which they relate. I will talk more about this on Wednesday.</p>
<p>The P&L is all about timing, estimates and assumptions. <em>Timing is everything</em>. And this is why the P&L is also subjective. As well as the fact that the accounting period isn’t related to actual cash flow and accruals etc, there is also the fact that many expenses are not clearly directly related to a sale so are difficult to allocate correctly.</p>
<p>This is why <em>profit is not cash</em>.</p>
<h2 id="depreciation-is-an-estimate">Depreciation is an estimate</h2>
<p>I <a href="/jfdi/finance-for-non-accountants.html">wrote about amortisation and depreciation</a> in my previous blog post. To reiterate: assets lose value over time, and this loss of value is representated as a cost on the P&L. Depreciation refers to tangible assets (e.g. office furniture, buildings) and amortisation refers to intangible assets (e.g. patents, trademarks, websites).</p>
<p>The value of the asset on the balance sheet is also reduced by the amount it’s been reduced by on the P&L, so it loses value over time. The current value of an asset on the balance sheet is called the ‘net book value’.</p>
<p>How depreciation is reflected on the P&L is a matter of what depreciation policy you adopt.</p>
<p>For example, it could be Straight Line depreciation – determine (or rather, guess) how long the asset will last and subtract an equal fraction each year. For example, if a laptop will probably last 4 years and costs £1,000, subtract £250 from the P&L every year for four years.</p>
<p>Another method is the Reducing Balance method, e.g. where you subtract a certain percentage of the remaining balance each year. (For the laptop, £250 in year 1. Then 25% of the remaining balance in year 2, so 25% of £750 – £187.50).</p>
<p>Note that there are several estimates being made here, as well as decisions about policy.</p>
<h2 id="the-order-of-the-pl-matters">The order of the P&L matters</h2>
<p>Start with: Sales/revenue</p>
<p>Deduct: cost of sales => this leaves gross profit</p>
<p>Deduct: operating expenses => this leaves operating profit/EBIT (EBIT = Earnings Before Interest and Tax)</p>
<p>Deduct: Interest => this leaves profit before tax (PBT)</p>
<p>Deduct: Corporation tax => this leaves profit after tax (PAT); also called net profit.</p>
<p>Note that if something is on a P&L it has an impact on profit, which can lead to poor decision making, <a href="/jfdi/finance-for-non-accountants.html">as I mentioned before</a>. For example, whether to build or buy software. Built software is an asset, on the balance sheet. Software you buy as a service is an operating expense, therefore reduces your profit. But building all the software you use is definitely not the right decision.</p>
<p>And a reminder: profit is an estimate. It will change, for example, if you change your estimate of depreciation. PROFIT IS NOT CASH.</p>
<h2 id="why-we-dont-like-ebitda">Why we don’t like EBITDA</h2>
<p>You may have heard the acronymn EBITDA. That doesn’t appear in the P&L above; EBITDA is Earnings before Interest, Tax, Depreciation and Amortisation.</p>
<p>It’s a useful way of comparing companies’ relative operating efficiency, and means you don’t get bogged down in different depreciation policies. They are also subjective, so it’s nice to be able to ignore them. However, those companies will still have to pay depreciation and amortisation, and it is a big cost. (<a href="https://en.wikipedia.org/wiki/Warren_Buffett">Warren Buffett</a> does not like EBITDA and is credited with saying “Does management think the tooth fairy pays?”).</p>
<p>Private Equity companies like talking about EBITDA because depreciation and amortisation take place over time, and private equity companies don’t plan to be around for the long term; the company will be bought and sold within 5 years so who cares if they invest for the future or not?</p>
<p>As well as Warren Buffett, the IoD, and my instructor on the course are not fans of using EBITDA as a measure, because amortisation and depreciation are real costs.</p>
<p>A better measure is free cash flow, which I will talk about shortly.</p>
<h2 id="the-cash-flow-statement-shows-where-funding-has-come-from-and-how-it-is-used">The cash flow statement shows where funding has come from and how it is used</h2>
<p>The third statement it’s important to understand is the cash flow statement. This was only introduced, for larger companies, in 1991.</p>
<p>There is nothing on the cash flow that cannot be calculated from the balance sheet, the P&L and the notes to the accounts, but it’s extremely valuable for understanding where the cash is coming from and going to, and what the actual position of the company is. The cash flow statement illustrates how funding has been generated and how it is used in the organisation.</p>
<p>The most important factor for an investor when deciding whether to invest in a company is management credibility. What are you going to do next to improve the company and make money for your investors? And one way to assess management credibility is looking at the cash flow. Does the cash flow match up to your strategy?</p>
<h2 id="free-cash-flow-is-how-much-cash-there-is-for-discretionary-spending">Free cash flow is how much cash there is for discretionary spending</h2>
<p>That is how much cash there is, less operating expenses. That is, operating cash flow, adjusted to reflect:</p>
<ul>
<li>interest paid</li>
<li>tax paid</li>
<li>capital expenditure</li>
<li>loan repayment arrangements</li>
</ul>
<p>Free cash flow shows the true cash generation of the business, and allows you to assess:</p>
<ul>
<li>the impact of future capital projects</li>
<li>repayment capacity for new loans</li>
<li>whether dividends should be distributed</li>
</ul>
<p>As a director, you need to know what your free cash flow is so that you can make strategic decisions.</p>
<p>Our course instructor said:</p>
<div class="quote">
Revenues: vanity<br />
Profits: sanity<br />
Cash: reality<br />
</div>
<h2 id="balance-sheet-analysis-quiz">Balance sheet analysis quiz</h2>
<p>At this point in the course, we did a very fun exercise where we were shown the balance sheets of 6 companies, with a few notes about them, and had to work out which was which, thinking about what their business model was, what would be dominant on their balance sheet, and what proportion of other things would be on the balance sheet.</p>
<p>There is no way before this course I would have had the first clue of where to start, but our group got this all correct. Hopefully I have given you enough information above to have a go at this yourself, so here is the exercise. I’ll be back on Wednesday with the answers, and part 2 of my write-up.</p>
<ol>
<li>Barclays Bank</li>
<li>Facebook</li>
<li>easyJet</li>
<li>The IoD (Institute of Directors)</li>
<li>Morrison’s supermarket</li>
<li>Apple</li>
</ol>
<p>Notes</p>
<ul>
<li>The IoD does not own its buildings but has leasehold interests. Delegates attending courses pay upfront for courses to be attended over time.</li>
<li>Like other banks, Barclays shows derivatives on both the assets and liabilities sides of its balance sheet.</li>
<li>Facebook completed the acquisition of WhatsApp for $22bn in 2014. WhatsApp was founded in 2006, had annual revenues of $30m and was loss-making. Facebook continues to be acquisitive. Goodwill is only very marginally amortised. It continues to be a highly profitable business and has yet to pay a dividend.</li>
<li>Apple started paying regular dividends in 2012 and has instigated a very active share buyback programme to return cash to shareholders. Borrowings were raised for this to take advantage of very low interest charges and to minimise their tax obligations. A high percentage of its profits are held outside of the US.</li>
<li>All the balance sheets are in percentages, not absolute size so you can’t tell which is which from that.</li>
<li>An empty cell indicates they do not have that kind of thing (or it’s so tiny as to be negligible)</li>
</ul>
<table class="underline-table">
<tr><th>Uses: Assets %</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th></tr>
<tr><td>Tangible/fixed assets</td><td>11</td><td>74</td><td>0.2</td><td>26</td><td>63</td><td>25</td></tr>
<tr><td>Intangible assets</td><td></td><td> 4 </td><td> 0.7 </td><td> 4 </td><td> 7 </td><td> 20 </td></tr>
<tr><td> Investments (long-term) </td><td> 31 </td><td> 8 </td><td> 28.1 </td><td> </td><td> </td><td> </td></tr>
<tr><td> Inventory/stock </td><td> 1 </td><td> 7 </td><td> </td><td> </td><td> </td><td> </td></tr>
<tr><td> Receivables/debtors </td><td> 7 </td><td> 3 </td><td> 29 </td><td> 19 </td><td> 4 </td><td> 8 </td></tr>
<tr><td> Cash & short-term investments </td><td> 30 </td><td> 3 </td><td> 22 </td><td> 51 </td><td> 19 </td><td> 42 </td></tr>
<tr><td> Other </td><td> 20 </td><td> 1 </td><td> 20 </td><td> </td><td> 7 </td><td> 5 </td></tr>
<tr><td></td><td>100 </td><td> 100</td><td> 100 </td><td> 100 </td><td> 100 </td><td> 100 </td></tr>
</table>
<p><br /></p>
<table class="underline-table">
<tr><th>Sources: Liabilities & equity %</th><th>1</th><th> 2</th><th> 3</th> <th> 4 </th><th> 5</th><th> 6</th></tr>
<tr><td>Payables/creditors <1yr </td><td> 31 </td><td> 33 </td><td> 46</td><td> 61</td><td> 32</td><td> 7 </td></tr>
<tr><td> Payables/creditors >1yr </td><td> 42 </td><td> 20 </td><td> 29 </td><td> 10 </td><td> 31 </td><td> 6 </td></tr>
<tr><td> Other </td><td> </td><td> </td><td> 19 </td><td> </td><td> </td><td> </td></tr>
<tr><td> Equity/shareholders' funds </td><td> 27 </td><td> 47 </td><td> 6 </td><td> 29 </td><td> 37 </td><td> 87 </td></tr>
<tr><td></td><td>100 </td><td> 100</td><td> 100 </td><td> 100 </td><td> 100 </td><td> 100 </td></tr>
</table>
<h2 id="answers-on-wednesday">Answers on Wednesday</h2>
<p>The answers to the quiz and more on accountancy principles, project appraisal techniques and other interesting financial concepts to follow!</p>
<p><em>This is post one of three posts. You can read <a href="/jfdi/finance-part-2.html">part two here</a> and <a href="/jfdi/finance-part-3.html">part three here</a>.</em></p>
Creating, defining and refining an effective tech strategy2021-02-26T00:00:00+00:00https://www.annashipman.co.uk/jfdi/tech-strategy-panel.html<p>Last month I <a href="https://leaddev.com/technical-direction-strategy/creating-defining-and-refining-effective-tech-strategy">facilitated a panel at Lead Dev Live</a> about how to build an effective tech strategy.</p>
<p>The panel were absolutely excellent: <a href="https://twitter.com/randyshoup">Randy Shoup</a>, VP Engineering and Chief Architect, eBay, <a href="https://twitter.com/_anandmariappan">Anand Mariappan</a>, VP Engineering, Impira, <a href="https://twitter.com/nasthagiri">Nimisha Asthagiri</a>, Engineering Director and Chief Architect, edX and <a href="https://twitter.com/OssomDade">Papanii Nene Okai</a>, Director of Engineering and Architecture, Venmo.</p>
<p>They all had a lot of great experience and insight to share. We talked about what a good tech strategy looks like, how to craft one, how to articulate and communicate it, and how to adapt it as times change. We also discussed the tricky problem of how to make sure people are following it. I could have talked to them for ages.</p>
<p>I made several notes during the session. Nimisha had fascinating insights and models about how to collaborate on a strategy with a distributed team, as she oversees an open source platform with a global community. Randy had brilliant thoughts on what a tech strategy should be, including actionable, pragmatic and falsifiable. Anand had great explanations of the importance of showing, rather than telling, to get buy-in for your tech strategy; and Papanii shared the extremely useful CBTI paradigm (Customer, Business, Team, Individual) for the order in which you think about whether something has value.</p>
<p>I asked the panel to summarise at the end what <em>the</em> most important thing for a tech strategy was, and we agreed:</p>
<ul>
<li>Know what problem you are trying to solve</li>
<li>Show don’t tell</li>
<li>Enable and empower decentralised decision-making</li>
</ul>
<p>The discussion afterwards in the LeadDev Slack was also really interesting. Someone asked for recommended reading on how to create a good strategy, and the following suggestions were made, all worth a read:</p>
<ul>
<li><a href="https://www.paperplanes.de/2020/1/31/on-drafting-an-engineering-strategy.html">On drafting an engineering strategy</a></li>
<li><a href="https://sarahtaraporewalla.com/agile/design/architecture/Defining-a-Tech-Strategy">Defining a tech strategy</a></li>
<li><a href="https://lethain.com/engineering-strategy/">Engineering strategy</a></li>
<li>and my blog post on how we created our tech strategy <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">The difficult teenage years: Setting tech strategy after a launch</a></li>
</ul>
<p>You can <a href="https://leaddev.com/technical-direction-strategy/creating-defining-and-refining-effective-tech-strategy">watch the panel here</a> (you need to register with LeadDev to access; this is free).</p>
Some notes on inclusive leadership2021-01-06T00:00:00+00:00https://www.annashipman.co.uk/jfdi/inclusive-leadership.html<p>I recently went on an inclusive leadership course at work. It was very interesting. There were four particular lists that I wanted to make a note of to refer back to:</p>
<h2 id="do-these">Do these</h2>
<ul>
<li>Be bold about getting names right</li>
<li>Start conversations about culture</li>
<li>Explore how your culture affects others</li>
<li>Learn to tell people of different ethnic groups apart</li>
<li>Get a cultural calendar</li>
<li>Be transparent about cultural norms and expectations</li>
<li>Do unconscious bias training to find out about your implicit associations</li>
<li>Get your staff trained to find their voice</li>
</ul>
<h2 id="dont-do-these">Don’t do these</h2>
<ul>
<li>Don’t ignore cultural difference</li>
<li>Don’t place an emphasis on “fitting in”</li>
<li>Don’t tolerate cultural norms that are at odds with values</li>
<li>Don’t try to bleach manifestations of faith from the workplace</li>
<li>Don’t prevent staff from attending affinity groups in work time</li>
<li>Don’t try to stop minority groups using their community languages at work</li>
<li>Don’t invest resources in leaders at the expense of minoritised groups</li>
<li>Don’t forget to acknowledge “the other job” some people do</li>
</ul>
<h2 id="what-makes-us-more-likely-to-act-on-our-unconcious-bias">What makes us more likely to act on our unconcious bias?</h2>
<p>This was really interesting, very closely linked to the two systems concept in <a href="https://uk.bookshop.org/books/thinking-fast-and-slow/9780141033570">Thinking, Fast and Slow</a> and also the idea of “taking deliberate action” as a mechanism for preventing mistakes, in <a href="https://uk.bookshop.org/books/turn-the-ship-around-a-true-story-of-building-leaders-by-breaking-the-rules/9780241250945">Turn the Ship Around!</a>.</p>
<p>Here are some of the things that make us more likely to be biased in our actions:</p>
<ul>
<li>Conflicting priorities</li>
<li>Deadlines</li>
<li>Need for closure/quick decisions</li>
<li>Being challenged</li>
<li>Anxiety</li>
<li>Low blood sugar</li>
<li>Cold</li>
</ul>
<p>It’s always worth taking a bit more time over decisions, taking a break, a walk round the block, or having a cup of tea and a biscuit!</p>
<h2 id="some-examples-of-manners-leading-to-bias">Some examples of ‘manners’ leading to bias</h2>
<p>She gave us some really good examples of how something that in one culture we might perceive as “good manners” are non-existent, or pereceived as bad manners in other cultures; something to be aware of when you are making snap judgments about people.</p>
<p>For example:</p>
<ul>
<li>eye contact means honesty</li>
<li>it’s not polite to talk about money</li>
<li>it’s rude to get too physically close</li>
<li>please form a queue</li>
<li>boasting is unseemly</li>
<li>one should always say please</li>
</ul>
<p>Reading this list, I can really feel how acting differently to some of those would provoke a reaction in me that is actually completely unjustifed as people have very different cultural norms.</p>
<h2 id="a-course-like-this-is-worth-doing">A course like this is worth doing</h2>
<p>These are the lists I wanted to make a note of for my own future reference, though they represent only a small fraction of what was covered in the three hour workshop. I don’t plan to write the rest of the course up. It’s definitely worth seeking out such a course yourself if you can.</p>
How Durable Teams Take You Further, Faster2020-12-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/durable-teams-keynote.html<p>Last week I gave a keynote at <a href="https://conference.ctocraft.com">CTO Craft conference</a> based on my <a href="https://medium.com/ft-product-technology/unlocking-value-with-durable-teams-a70efb435a19">blog post about moving to durable teams</a>.</p>
<p>It was an outline of why and how we made the move, and my reflections on how it’s gone, one year on. The <a href="https://www.slideshare.net/annashipman/how-durable-teams-take-you-further-faster">slides are on Slideshare</a>, and <a href="https://vimeo.com/showcase/7882845/video/488034792">video of the talk is on Vimeo</a>.</p>
Turn the Ship Around!2020-11-26T00:00:00+00:00https://www.annashipman.co.uk/jfdi/turn-the-ship-around.html<p>Prompted by some discussions at work, I recently read <a href="https://uk.bookshop.org/books/turn-the-ship-around-a-true-story-of-building-leaders-by-breaking-the-rules/9780241250945">Turn the Ship Around!</a> I enjoyed it. Here are my notes.</p>
<h2 id="making-followers-into-leaders-in-the-navy">Making followers into leaders in the Navy</h2>
<p>I had previously seen a talk about the concepts by the author, L. David Marquet. Recently, one of my team shared a link to this <a href="https://www.youtube.com/watch?v=OqmdLcyES_Q">short video (10 min)</a> when we were having a discussion about how to take more ownership, and I thought I would read the book to get the practical steps.</p>
<p>The context is fascinating, because being a leader at all levels rather than following orders, is expressly something I do not associate with the military.</p>
<p>The book is essentially a case study of what L. David Marquet and his team aboard a nuclear submarine did to move from a “leader-follower” approach to a “leader-leader” approach.</p>
<p>Each chapter illustrates an idea with an example, which makes it very easy to read, and then supplements it with questions you can use to think about how this idea could work in your own organisation. I always find it much more powerful when a business book is written by someone who actually has the experience of implementing the ideas they write about (like <a href="/jfdi/high-output-management.html">High Output Management</a>, my favourite).</p>
<h2 id="the-goal-was-leadership-at-all-levels">The goal was leadership at all levels</h2>
<p>The end result was a submarine where each crew member took responsibility for their own area. Instead of the Captain making decisions, crew members would come to him and say “Captain, I intend to…” and he would say “very well”.</p>
<p>And this was just the decisions he needed to know about. 95% of the decisions on board were taken without his input.</p>
<p>The fact that they had leadership at all levels also meant they were more resilient. They had a wide pool of talent. He gives an example of wanting to let two senior people have extended leave for personal reasons at the same time, which he was able to do because there were others able to do those roles.</p>
<h2 id="he-didnt-know-the-ship-so-had-to-be-curious-rather-than-rote-questioning">He didn’t know the ship, so had to be curious (rather than rote questioning)</h2>
<p>He was in an interesting position in that he had trained for a different type of submarine, so when he joined this post he was not as technically competent as he was used to being (and ask the Navy traditionally expect leadership to be).</p>
<p>This meant when he asked the crew questions about the equipment it wasn’t to test them on their knowledge but because he genuinely did not know. Eventually he became fully technically competent on the ship but by then the patterns of interaction had been set.</p>
<p>He talked about his own struggles with wanting to be the smartest person in the room, and I found this very insightful; one of the challenges with a “leader-leader” approach is having to let go of your idea of yourself as someone who has all the answers, and this can be very hard.</p>
<p>I liked this, because it ties into my experience joining as Technical Director of a product that I didn’t build, so could not hope to be anywhere near as technically competent with as the people who built it and work on it every day.</p>
<p>One of the questions he poses at the end of the section on this is “Is technical competence personal or organisational?”</p>
<h2 id="he-met-each-member-of-the-crew-and-asked-them-some-questions">He met each member of the crew and asked them some questions</h2>
<p>One of the first things he did when taking command of the submarine was meet with each crew member and ask them a set of questions very similar to <a href="/jfdi/meeting-everyone.html">the ones I asked my team when I joined</a>. (One question I might add for next time is “What is the best thing I can do for you?”)</p>
<p>I found that very interesting, because that idea was recommended to me by a previous mentor; he had been recommended it by a mentor of his, who was a Rear Admiral in the Navy, so I wonder if there was a direct connection here!</p>
<h2 id="emancipated-not-empowered">Emancipated, not empowered</h2>
<p>He talks a lot about how “empowerment” doesn’t work. It’s the leader allowing the crew to be empowered; so it’s still “leader-follower”, which drowns out the message. Instead, he talks about “emancipation”.</p>
<p>He says he wanted to make sure his team deliberately decided to take charge. “It wouldn’t be any good if I directed them. You can’t invoke leader-follower rules to direct a shift from leader-follower to leader-leader.”</p>
<p>When it’s going well you no longer have the ability to empower your staff, because they are no longer relying on you for their source of power.</p>
<p>He then defines leadership as communicating people’s worth and potential to people so well they are inspired to see it themselves.</p>
<h2 id="people-need-to-own-the-consequences-of-decisions-they-make">People need to own the consequences of decisions they make</h2>
<p>He started small, with each department head in charge of authorising leave. But then they weren’t in charge of the consequences, e.g. if too many people were on leave at once and there weren’t enough people to cover the watch. So then each department head was in charge of leave and the watch rota; i.e. the consequences of their decisions.</p>
<p>This became: each department head monitors their own departments and are responsible for getting the work done. (This works well when there are clear domains; one challenge we have on our team is that we share responsibility for a large domain rather than being able to split it clearly by area. This is something we’ll have to figure out.)</p>
<p>He requires each department head – and eventually, each crew member – to own both problems and the solutions to them.</p>
<p>Delegating authority in this way means always leaving room to question orders, and also people being able to tell him he is wrong. He recounts a story of when an officer much junior to him tells him, in response to an order, “No, Captain, you’re wrong.” The junior officer was correct, and had they instead followed the Captain’s orders, they would not have achieved their mission.</p>
<h2 id="i-intend-to">I intend to…</h2>
<p>Initially, they would come to him and say “I intend to…” and he would ask questions, for example, have you considered this safety aspect. Then he started asking them what he was thinking. Ultimately, instead of him asking questions, his crew would give enough information so that he just needs to assent.</p>
<p>In order to do this, you need to think like your boss to understand what questions they may have. He says on his submarine, they had no need of leadership development programmes – the way they ran the ship <em>was</em> a leadership development programme.</p>
<p>While reading this I wondered how we could apply it. My <a href="/jfdi/delegating-to-a-team.html">aim is to have the principal engineers on my team running the team without me</a>, so something like this would be good. One of the challenges is that the tasks are much larger or longer-lived: it’s not “I intend to submerge the ship”, it’s more like “I intend to spend three weeks working on a proposal for how to make our content stores more consistent and cost-effective” and so would have to involve information about why that was a priority compared to other tasks.</p>
<p>It also made me think about the importance of clarity around what done looks like. In the first case, ‘done’ is the ship is safely suberged. What is ‘done’ in the second case? A case study? A recommendation? The actual work – consistency among our content stores?</p>
<p>However we do it, this proactive communication is really valuable. Rather than the leader holding all the threads and chasing for updates, each person owns the thread and proactively communicates the progress and goals.</p>
<h2 id="how-can-we-delegate-authority-further">How can we delegate authority further?</h2>
<p>He suggests thinking specifically about what mechanisms/processes can be changed to delegate authority further. What information, context, understanding, tools do people at each level need in order to have authority?</p>
<p>He describes an exercise. Identify where excellence is created in your company, e.g. at interfaces with the customer. Figure out what decisions the people at that interface need in order to achieve excellence. Finally, understand what it would take to get those employees to be able to take those decisions. Generally, this will be technical competence, thorough understanding of the organisation’s goals, authority, and responsibility for the consequences of the decisions made.</p>
<p>I like the idea of asking people, what would you need in order to be able to make this decision rather than passing it up the chain? And then figuring out what we can change so that they have that.</p>
<p>He also asks you to think about: what do I, as a proponent of the leader-leader approach need to delegate to show I am willing to walk the talk? Good question. It reminds me of some excellent advice I got years ago on first taking on a leadership role: “Delegate everything. Far more than feels comfortable.”</p>
<h2 id="as-authority-is-delegated-learning-at-all-levels-becomes-more-important">As authority is delegated, learning at all levels becomes more important</h2>
<p>If people need to make their own decisions, they need to be technically competent in their area.</p>
<p>They created a creed – ultimately, it was about learning (not ‘training’, they felt that was too passive).</p>
<p>I loved this bit of their creed: <em>“Our vision of our command is a learning and competence factory. The raw materials are the new personnel reporting aboard each week, new equipment, and tactics. The product is well-qualified, experienced sailors who, upon detaching from the command, carry their competence throughout the Navy. Each of you, then, is both a product of the factory (when you learn) and a machine in the factory (when you help others learn).”</em></p>
<p>It is about learning while doing. “Instead of looking at a task as just a chore, look at it as an opportunity to learn more about the associated piece of equipment, the procedure, or if nothing else, about how to delegate or accomplish tasks.”</p>
<p>Finally, to each crew member’s question, ‘what do you expect me to do?’: “I expect you to be a better submariner each day.”</p>
<h2 id="this-revolutionised-how-they-thought-about-training-programmes">This revolutionised how they thought about training programmes</h2>
<p>Rather than passive absorbing of information, their training programme was actually an enabler that allowed them to pass decision-making authority to lower and lower levels.</p>
<p>His suggestion about how to apply this in your own organisation is to think about the sentence completion “Our company would be more effective if [level] management could make decisions about [subject].” (As with all his exercises, this is not you on your own, but you with your leadership team, or at a company off-site).</p>
<p>Then “What, technically, do the people at this level of management need to know in order to make that decision?” This leads to a list of topics for training, and “you can directly connect the training topics to increased employee decision making and control – in a word, empowerment.”</p>
<p>He found the learning mindset helped his perspective too, allowed him to approach inspections calmly, even eagerly, rather than anxious about the result; and this impacted the crew as well, who sensed this.</p>
<h2 id="it-also-revolutionised-the-way-they-thought-about-inspections">It also revolutionised the way they thought about inspections</h2>
<p>Because they created an atmosphere of learning and curiosity among the crew, inspections were welcomed rather than something to greet with defensiveness. The crew would start by handing the inspectors a list of deficiencies so that they are on the agenda and will get resources to solve them. While inspectors were looking round the ship, crew would ask things like “I’m having a problem with this. What have you seen other ships do to solve it?”</p>
<p>He asked how do we use outside groups, the public, social media comments and government audits to improve our organisation? What are the costs of being open about problems in your organisation?</p>
<h2 id="context-and-clarity-at-all-levels-also-becomes-more-important">Context and clarity at all levels also becomes more important</h2>
<p>As the level of control is divested it becomes more and more important that the team be aligned with the goal of the organisation. If clarity of purpose is misunderstood then the criteria by which a decision is made will be skewed and suboptimal decisions will be made.</p>
<p>Don’t move information to authority, move authority to where the information is. “If you ask people what authorities they would like in order to make their jobs easier you’ll definitely get some ideas.”</p>
<h2 id="short-early-conversations-are-good">Short early conversations are good</h2>
<p>Short early conversations are useful – not telling people what to do but an opportunity for them to get early feedback on how they are tackling problems. The demand for perfect products first time see you them is a waste of subordinates’ time. The idea is to protect executive time, but it wastes more time overall.</p>
<p>Early conversation can unearth well-meaning but erroneous translation of intent.</p>
<h2 id="think-out-loud-both-superiors-and-subordinates">Think out loud (both superiors and subordinates)</h2>
<p>He talks about this as a mechanism for sharing context both ways. On their submarine there is a constant buzz of informal communication, and he suggests you walk around to get a view of how much is communicated informally – in the current climate, I guess that would equate to paying attention in Slack.</p>
<p>When his crew didn’t think out loud, that was when he was tempted to step in and interfere, because he didn’t know things were heading in the right direction. When he was thinking out loud, he was both imparting useful context, and also modelling that “lack of certainty is strength and certainty is arrogance.”</p>
<p>The goal is to create an environment in which people express their uncertainty and fears as well as their innovative ideas and hopes.</p>
<h2 id="as-a-leader-dont-make-decisions-on-your-own">As a leader, don’t make decisions on your own</h2>
<p>Emergency situations require orders but few situations are actual emergencies.</p>
<p>Take time to let others respond to the situation. Create space for open decisions.</p>
<p>If issues that require decisions frequently come up on short notice, then you have a reactive organisation locked in a downward spiral. “When issues aren’t foreseen, the team doesn’t get time to think about them; a quick decision by the boss is required, which doesn’t train the team, and so on.”</p>
<p>He suggests a few ways to deal with it. If it’s genuinely urgent, make the decision then ask the team to “red-team” the decision and evaluate it. If it needs to be made quite soon then ask for input then make the decision. If it can be delayed, then get input from the team (though do not attempt to come to consensus: “that results in whitewashing differences and dissenting votes. Cherish the dissension. If everyone thinks like you, you don’t need them.”)</p>
<h2 id="resist-the-urge-to-provide-solutions">RESIST THE URGE TO PROVIDE SOLUTIONS</h2>
<p>I’ve put that in capitals both because he talks about it a lot, and because I feel that urge a lot! It reminds me of one of the most important recommendations in <a href="/jfdi/what-got-you-here.html">What got you here won’t get you there</a> – “Stop adding too much value.”</p>
<h2 id="change-behaviour-before-mindsets">Change behaviour before mindsets</h2>
<p>He had an interesting idea about acting as if you have the organisation you want, in the same way that if you smile, you start to feel better. “Instead of trying to change mindsets and then change the way we acted, we would start acting differently and the new thinking would follow.”</p>
<p>His suggestion for how to implement this: “I’d know we’d achieved this cultural change if I saw employees [doing something specific]”. Then implement the doing of the something specific.</p>
<p>This is an interesting idea; I’m not sure it would work in all cases but I can see the benefit of, for example, requiring that everyone is polite on pull requests as a rule, instead of waiting until everyone has the mindset that everyone is deserving of respect.</p>
<h2 id="how-to-set-long-term-goals">How to set long-term goals</h2>
<p>He says “begin with the end in mind” (which he notes is from the book ‘7 Habits of Highly Effective People’).</p>
<p>One way he implemented this was by having mentoring meetings with his staff, where the rule was they could only talk about long term issues, not operational issues. He asked them to write their ‘end of tour awards’ (3 years out) or – if too far away to think about – then their next year’s performance evaluation.</p>
<p>He got them to make it specific and measurable by asking questions like “how would you know if this had been achieved?” And digging deeper: e.g. if it was “we’d have fewer critiques”, the next question is how many critiques did you have last year? In the process they often learned that they hadn’t been keeping track of appropriate data. It meant the end of year reviews had concrete data in them – instead of “reduced significantly” it would say, e.g. “reduced by 43%”.</p>
<p>He asks “for how far in the future are you optimising your organisation?”</p>
<h2 id="success-is-measured-by-what-happens-after-you-leave">Success is measured by what happens after you leave</h2>
<p>Something that really resonated with me is how he talks about your legacy as a leader. He notes that a lot of ships would be described as a “good ship” but then the captain moves on and they become a “bad ship”. He correctly notes that this means the captain is the one making this happen (and gives examples of captains working really long hours, running around to make sure every job is being done the way they’d like). If you leave, and the organisation collapses behind you then you have not truly built a good organisation.</p>
<p>He can say his methods were successful because ten years later they can really “assess the true success of the work – continued operational excellence and implausibly high promotion rates”.</p>
<p>The problem for him is the same as for me: we don’t evaluate leaders’ effectiveness after they leave but while they are there, so the incentive is to focus on short-term results at the cost of long-term sustainability.</p>
<p>To promote long term success he had to ignore the short term reward system. He describes this as “caring but not caring – caring about subordinates and the organisation but little about the organisational consequences to yourself.”</p>
<h2 id="there-are-more-simliarities-between-the-navy-and-software-development-than-id-realised">There are more simliarities between the Navy and software development than I’d realised!</h2>
<p>Some points he made resonated so much! For example, he said on ship there is a “constant tension between doing things right and meeting deadlines” – I shouldn’t be surprised at these parallels but I was.</p>
<p>He also talked about a process that I recognise as a <a href="https://codeascraft.com/2012/05/22/blameless-postmortems/">blameless post-mortem</a> or retrospective, though he didn’t call it that; they created a culture where they were open about mistakes and looked for ways to prevent or mitigate them rather than one where they wasted time and effort trying to avoid problems. He found when he arrived that the crew’s actions were motivated by “avoiding problems” e.g. following a checklist, pleasing an inspector, looking good.</p>
<p>He also talks about replacing general terminology with clear, concise, specific directions – avoiding jargon and clarity in communication is something very close to my heart.</p>
<h2 id="he-is-honest-about-the-challenges">He is honest about the challenges</h2>
<p>Unlike a lot of leadership books, he acknowledges the challenges, things that went badly, and where this may be hard for you and was hard for him.</p>
<p>For example, a few times, he talks about situations when he didn’t do what he knew he should. One time he says he was too tired to engage in the discussion and just gave an order. “A more enlightened apporach would have been to engage in a discussion… that’s what I wanted to do, but I just didn’t have the energy or time anymore. All day, every day, it seemed like that’s all I did. It was tiresome.”</p>
<p>He notes that when tired or under pressure, he fell back on bad habits. This was something I really noticed in March, at the beginning of the pandemic – I and most of my colleagues reverted to our default behaviours, which are often not the behaviours we’ve worked hard on to build our teams and good working relationships.</p>
<p>He points out that clear feedback is crucial: “taking care of people doesn’t mean shielding them from the consequences of their own actions” but also admits that something he struggled with all his career was “balancing the courage to hold people accountable for their actions with my compassion for their honest efforts”.</p>
<h2 id="the-most-important-control-is-control-over-yourself">The most important control is control over yourself</h2>
<p>Summing up, he points out that the control you need is “the self-control to give control”, which creates leaders. He says “rejecting the impulse to take control and attract followers will be your greatest challenge and, in time, your most powerful and enduring success.”</p>
<h2 id="this-books-is-worth-reading-whatever-your-level-at-work">This books is worth reading whatever your level at work</h2>
<p>I made A LOT of notes and have only covered some of it here; for example he talks about increased authority of chiefs increasing retention among more junior staff, about using principles effectively to guide decision-making and about “deliberate action” as a mechanism for preventing mistakes which happen on autopilot; and much more. So I really recommend reading it yourself.</p>
<p>It’s not just for those in a leadership position; he suggests that people at the front line might want to read it to embrace decision making, and/or challenge leaders.</p>
<p>I really enjoyed this book because it completely reflected what I think of as good leadership; building a strong team that can run without you, growing and developing the people who report to you, building resilience and an emancipated team who can act to get to the right outcomes without needing to wait for instruction or permission.</p>
<p>It offered some really practical steps to help me get there, and some good challenges to where I might think I am doing it already but have a lot of room for improvement.</p>
How to sell2020-10-01T00:00:00+00:00https://www.annashipman.co.uk/jfdi/selling.html<p>Even if we don’t work in sales, we often need to sell something: an idea, a course of action, or ourselves (e.g. in a job interview). It’s a useful skill to have in your toolbox. I recently had a very useful conversation with the excellent <a href="https://twitter.com/sladejon">Jon Slade</a>, who is the Chief Commercial Officer at the Financial Times, and he outlined some sales techniques.</p>
<h2 id="let-me-take-away-that-pain-you-feel">Let me take away that pain you feel</h2>
<p>When I wrote about my team’s tech strategy, <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">No next Next</a>, I wanted to include the commercial aspect so that it was clear how it was relevant to the business’s goals. I talked about how it took two years and cost £10m to build the new site. Our tech strategy is about making the new site sustainable, so in a few years we don’t have to spend another £10m and another two years rebuilding it.</p>
<p>I was quite proud of demonstrating the commercial angle, but when I discussed it with Jon, he said that, from a pure sales perspective, it wasn’t very compelling. He described it as “let me take away that pain that you don’t feel”.</p>
<p>When expressed this way, this made perfect sense! A better formulation would be “Let me take away that pain that you <em>do</em> feel”; or “Let me give you this thing you want.”</p>
<h2 id="whats-the-pain">What’s the pain?</h2>
<p>I found it was a really useful exercise when thinking about how to pitch an idea to think about what pain I could take away, or what thing I could deliver that they want. This means you are then selling something you know they need.</p>
<p>What’s the pain for the people I am pitching to? How can I take it away? How can I make that clear?</p>
<p>Jon said “Good solutions unburden the buyer”.</p>
<h2 id="dipada">DIPADA</h2>
<p>It can help to use a framework for a sales pitch. One Jon mentioned is DIPADA.</p>
<p>Described in <a href="https://mutiny.asia/insights/sales-library/sales-article/dipada-simple-frameworks-for-perfect-pitches">this article about sales conversations</a> as a “tortured but popular mnemonic”:</p>
<ul>
<li><em>Define</em> the problem</li>
<li><em>Identify</em> the solution</li>
<li><em>Prove</em> your capability</li>
<li><em>Agree</em> an outcome</li>
<li>Stimulate <em>Desire</em></li>
<li>Drive <em>Action</em></li>
</ul>
<p>What I really like about this is that the first three, DIP, map very well onto <a href="/jfdi/good-strategy-bad-strategy.html">what I have learned about strategy</a>: strategy is diagnosis, a vision and the steps to get there.</p>
<p>To give the example of a job interview: Define the problem, identify yourself as the solution, prove you can do it. Here is a very contrived example, “I tried using a screen-reader on your website before this interview, and I found it didn’t work. That’s something I’d love to help with; I have a strong background in building accessible sites, and as you can see from my CV my previous role was as a front-end accessibility engineer.”</p>
<h2 id="whats-your-elevator-pitch">What’s your elevator pitch?</h2>
<p>This is something familiar to me from planning talks. What’s the one thing you want people to take away from your talk? Or your pitch to the board to invest in some work? Or your change of direction to your team?</p>
<p>For a job interview, it might be a summary of your main selling point. To use the previous example, you might want to leave them with “I will fix your accessibilty problem”. Jon called this your calling card.</p>
<p>I think this is a useful tool for everyday, not just pitches. What do you want people to think of you?</p>
<h2 id="dont-talk-about-weaknesses-turn-them-into-strengths">Don’t talk about weaknesses, turn them into strengths</h2>
<p>Don’t talk about your own weaknesses. Don’t say “my company does not have many clients”, say “my company focuses deeply on all projects we take on”.</p>
<p>Everything can be turned around. Don’t say “your staff don’t know anything about the financial sector”, say “we have deep and extensive experience in finance and can help you make the right decisions”.</p>
<h2 id="include-a-price-tag">Include a price tag</h2>
<p>Numbers are good. They don’t have to be actual price tags; you can say “I will increase the productivity of your team by 25% in year one”, “I will acquire 4 new accounts per week”.</p>
<p>Numbers help with proving your competence; it anchors your pitch in the real world and what you can really offer, rather than the theoretical.</p>
<h2 id="be-confident-by-reminding-yourself-of-your-strengths">Be confident by reminding yourself of your strengths</h2>
<p>Your buyers also want to buy confidence. They want to feel you really believe you can deliver what you’re selling.</p>
<p>Jon suggested it might be a good idea, before a big pitch (or talk, or whatever) to write down good things about yourself, things you know you can do, and read them, or say them to yourself in the mirror. An executive coach I have been working with also suggested the same: it can be helpful to remind yourself of what you are good at, especially if you are nervous or feeling doubts.</p>
<h2 id="many-things-are-sales-pitches">Many things are sales pitches</h2>
<p>I found this advice extremely useful. I hadn’t considered any aspect of my job to be sales, but now I realise, lots of it is, and probably yours too.</p>
<h2 id="different-parts-of-the-business-have-a-lot-to-learn-from-each-other">Different parts of the business have a lot to learn from each other</h2>
<p>For learnings in the other direction, engineering to commercial, you should read Jon’s excellent post about <a href="https://medium.com/ft-product-technology/from-cco-to-tech-intern-my-week-of-work-experience-6dcb6adba5be">his internship in engineering</a>.</p>
Meeting everyone on a new team2020-09-17T00:00:00+00:00https://www.annashipman.co.uk/jfdi/meeting-everyone.html<p>When I joined the Financial Times as Technical Director for FT.com, I inherited a team of around 50 engineers. One of the first things I did was meet each of them for a one-to-one. I was initially resistant, but it was extremely valuable, I’m glad I did it, and I would definitely do it again in a future role.</p>
<h2 id="my-mentors-advice-about-the-content-of-the-meeting">My mentor’s advice about the content of the meeting</h2>
<p>The idea was suggested to me by a mentor, who’d been advised to do it by <em>his</em> mentor, a Rear Admiral, who said this was something you should do whenever you have a team of fewer than 150 people. My mentor gave me some tips:</p>
<ul>
<li>Be clear about whether you will take action or whether this is for information only.</li>
<li>This should mostly be about listening – you should talk for maybe 5 minutes and they should talk for 25.</li>
<li>It’s to find out what’s going well and what’s not going well.</li>
<li>It’s informal, but make sure it’s in an enclosed meeting room so that people feel they can speak freely.</li>
<li>Sometimes it will be quite boring, sometimes you may just learn a lot about someone’s family or hobbies, but that is useful from a getting to know people/relationship-building perspective and it means that you know some things about that person.</li>
<li>Aim is to get a bit about their background, their priorities and their pressures.</li>
</ul>
<h2 id="making-time-was-hard">Making time was hard</h2>
<p>I was initially resistant because of the time commmitment. With a team of ~50, that’s a lot of hours, and I was also working four days a week so each meeting takes up a greater proportion of time. However, once I’d made the decision to do this and announced my intention, it was important to me to follow through, so I made sure to make time.</p>
<p>I scheduled four of these 1:1s a week, starting with the people reporting directly to me and then on down the management chain.</p>
<h2 id="i-ran-each-meeting-in-the-same-way">I ran each meeting in the same way</h2>
<p>Firstly I ran through everything I planned to cover, and then stepped through it.</p>
<h3 id="introductory-comments">Introductory comments</h3>
<ul>
<li>I am asking the same questions to everyone</li>
<li>This is information for me only to get an idea of themes and how things are going; I’m not explicitly planning to take action on anything we discuss, so if something comes up that I need to take action on, let’s make sure we discuss that after this meeting</li>
<li>This is confidential. If you say something about someone else I’m not going to go and tell them. I may report on ‘what people are saying’, but I’ll say ‘the engineers feel’ or ‘an engineer said’; I won’t say “[Your name] said…”</li>
</ul>
<h3 id="what-were-going-to-discuss">What we’re going to discuss</h3>
<ul>
<li>First I’ll introduce myself, and tell you a bit about my background</li>
<li>Then, if you like, I’d love you to tell me a bit about yourself – as much or as little as you feel like sharing</li>
<li>Then we’ll discuss the following questions:</li>
</ul>
<ol>
<li>What do you think are the most important things we should be doing over the next year?</li>
<li>What will get in the way of us doing that?</li>
<li>What’s going well, i.e. what should we make sure we don’t change?</li>
<li>Is there anything you think I should know about?</li>
</ol>
<h2 id="is-there-anything-i-should-know-about">Is there anything I should know about?</h2>
<p>When I asked this question I talked a bit about why I was asking. I explained that I might not necessarily see or know things that may seem apparent to them, and while they should always feel able to bring things to me, now was a good opportunity to do so. It was an opportunity to make sure I’ve heard what’s important to you, what things should change and what things should stay the same.</p>
<p>This question always elicited very interesting responses, from organisational issues, to personal information people felt it was valuable for me to know about them.</p>
<h2 id="i-told-them-what-i-was-planning-to-ask-in-advance">I told them what I was planning to ask in advance</h2>
<p>I put all the information in the meeting invite.</p>
<div class="quote">
<p>I mentioned that I wanted to have a chat with everyone on FT.com to understand how things are going, does this time suit you for this?</p>
<p>The meeting agenda is the same for everyone; a quick intro and then the following questions (I'll go through this in the meeting too):</p>
<ul>
<li>What do you think the most important things we should be doing over the next year?</li>
<li>What will get in the way of us doing that?</li>
<li>What’s going well, i.e. what should we make sure we don’t change?</li>
<li>Is there anything you think I should know about?</li>
</ul>
<p>Thanks,</p>
<p>Anna</p>
</div>
<p>Some people did not read the meeting invite and came with no idea what the meeting was about. Some people had fully prepared and written notes that they then read out to me. Actually people having prepared sometimes was less useful, because sometimes it led the conversation to solutions rather than problems. However it was great that people had really given it some thought.</p>
<h2 id="making-notes-felt-too-much-like-a-promise">Making notes felt too much like a promise</h2>
<p>Each meeting was half an hour. In the very first one, I made notes in a notebook, but I realised that created an implicit commitment that I was going to take action on everything that was said, even though I had said it was information only.</p>
<p>However, I do not have a very good memory, so for all the subsequent ones I made a few notes after each meeting of key themes. This meant I couldn’t do more than two in a row or go straight into another meeting, so it made scheduling slightly harder. These days, people are much more aware of the shorter meeting approach so if doing this again, I’d go for the ‘therapy hour’ – 25 minutes for conversation then 5 minutes for me to make the notes.</p>
<h2 id="introducing-myself">Introducing myself</h2>
<p>In my intro, I gave a potted career history. Starting from my degree in philosophy, and my first career in <a href="https://www.barringtonstoke.co.uk/">children’s book publishing</a>, through teaching myself to code, my <a href="https://www.hw.ac.uk/study/uk/postgraduate/information-technology-software-systems.htm">masters in Software Systems</a> and then my 15+ year career in programming, infrastructure and operations, technical architecture, and my previous role as <a href="/jfdi/a-year-in-the-life-os-lead.html">Open Source lead</a>. I also talked about what appealed to me about the job as Technical Director at the FT.</p>
<p>I said roughly the same thing to everyone. I don’t normally introduce myself and give my background, but in this case I thought that as a new Tech Director most of them would not be working closely with me, and I would not be contributing code, so it was worth giving my credentials.</p>
<p>My mentor had suggested I also say something personal. I think he intended something like “married with two children” (or whatever), but instead, I tried to give a different kind of personal detail, something about my interests. I tried to come up with a different one for each conversation, for example something about my <a href="https://twitter.com/annashipman/status/1043917006477643777">cross-stitch hobby</a>.</p>
<p>This part was the hardest part for me, because prior to this I had generally enjoyed keeping a clear boundary between work stuff and personal stuff, so that definitely didn’t cover talking about cross-stitch, or my home life, on a first meeting. However, I had been trying to bring more of my personal self to work, and this part of the intro did lead to some really interesting conversations and I think helped make a better connection.</p>
<p>Of course, these days, when we are all at home, my personal life is in meetings with me, so it’s good I’d already started on that journey!</p>
<p>Giving so much information in my introduction also allowed the other person to introduce themselves how they wanted. Some talked career history, some focused on their hobbies, others were really open about their lives and aspirations.</p>
<h2 id="i-am-so-glad-i-did-this">I am so glad I did this</h2>
<p>My mentor was wrong about one thing – none of the conversations were boring.</p>
<p>In my first few months in the new job, I often felt really stretched for time, but I never regretted a single one of these meetings; it was always extremely interesting, my team are brilliant and it was great to meet them one on one, and each conversation always contained some valuable information.</p>
<p>There were two very valuable things about this for me.</p>
<p>The first was getting an idea of what change was needed. These meetings gave me a brilliant insight that wasn’t available elsewhere. Patterns started emerging very quickly, and formed the basis of our <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">tech strategy</a>.</p>
<p>The second was building relationships. A lot of the people I had 1:1s with I would not have come into contact with during the course of the ordinary working week. It would have taken time to meet everyone at socials, and it wouldn’t have been the same quality of conversation. I still feel, two years on, that I know a bit about all the people I had those conversations with, which has felt to me like a good foundation for our subsequent conversations.</p>
<p>It was also good, as someone who is a bit shy, to have names to faces quite quickly and people to say hello to when walking round the office.</p>
<h2 id="was-it-useful-for-my-team">Was it useful for my team?</h2>
<p>About a year later, I asked some of the people with whom I’d had these conversations whether they’d been useful (in an anonymous form).</p>
<p>All of the people who responded said they found the conversation valuable, and some of their comments were:</p>
<ul>
<li>“It broke down barriers and helped me feel less intimidated about approaching you, whether to talk about work or just to have a general chit chat. You are a very busy person who I wouldn’t ever work with directly so it was good to feel that you knew I existed.”</li>
<li>“There is hardly any opportunity for me to talk to people in higher position like you except when the team has a big problem. The 1:1 was really casual and I felt really comfortable talking to you. It was a good time to know what kind of person you are. If we didn’t do the 1:1, the answer of the question below “Do you feel able to raise issues with me?” would be “No”.”</li>
<li>“We sat down when you first started and it was nice to get some one-to-one time because it’s not often you get to do that with a Technical Director. It was nice to raise issues but for me, it was more of an opportunity to understand if I could trust you in the future with raising issues. Raising issues can be difficult and scary so it’s important to know if the person you are raising them to is receptive.”</li>
<li>“It really showed that you cared about us as humans, and how we fit with the rest of the team. It was also a great opportunity to get to know you”</li>
<li>“I think often of that conversation”</li>
</ul>
<h2 id="did-it-make-them-feel-more-able-to-raise-issues-with-me">Did it make them feel more able to raise issues with me?</h2>
<p>One of the challenges of being in a senior leadership position is that there are lot of things you just don’t see. Issues that are completely obvious to people in teams, doing the work, are just not clear or even visible to those in leadership positions.</p>
<p>There are various mechanisms to address this, like staff surveys, retrospectives, making sure line managers feel comfortable raising issues with their line managers/their line managers’ managers, etc, but it’s also useful to have that direct line open in case all others fail.</p>
<p>One of the questions I asked people in my follow up after a year was whether “do you feel able to raise issues with me?” and the answers ranged from “yes”, to “I did at the beginning after having this meeting but less so as time went on.”</p>
<p>This suggests that while a one-off is valuable and creates that openness and makes it clear you are interested in feedback and input from the team, that effect doesn’t last forever.</p>
<p>I think the relationship-building aspect does have a longer lifespan though; each small interaction you have with people builds on that.</p>
<h2 id="would-i-do-it-again">Would I do it again?</h2>
<p>I would definitely do it again when starting another job. It gave me an incredible head-start on understanding the areas that needed improvement, and let me start to get to know my impressive, excellent team.</p>
<h2 id="would-i-do-it-regularly">Would I do it regularly?</h2>
<p>One question I have worried over (and is the reason it’s taken me so long to write this blog post) is whether it should be a regular thing. My previous mentor said he did it annually.</p>
<p>It would be a very big time commitment to do annually, around 28 hours, which incidentally is my working week if I did nothing else at all. But I couldn’t get it done in a week, as I find back-to-back meetings too draining, particularly intense focus ones like this. It would take probably 6-8 weeks a year to get them all done, and that would involve prioritising it above most of my other commitments.</p>
<p>At this stage, I have a lot of ways of gathering information about how my team are doing. I have open conversations with direct reports who keep me in the loop. We have a Peakon survey, we do a Spotify healthcheck, we have a line managers channel where we raise issues, teams have retrospectives and people do reach out to me with issues. So while I am sure there are issues that I am completely missing, I am not sure (famous last words!) that there are enough to justify the time again.</p>
<h2 id="talking-to-joiners-and-leavers">Talking to joiners and leavers</h2>
<p>While I’ve decided not to make this a regular thing, I still want to know everyone in my team. What I do now is have a 5–10 minute conversation with new starters in their first week or so, so the initial barrier of a conversation with me is broken.</p>
<p>I also have a coffee with people when they leave the team or the FT. That is another great opportunity to ask the “is there anything I should know” question and get often very useful insights.</p>
<p>Now that we are all working from home, I arrange occasional catch ups* with people on my team to see how they are doing, and to try and replace the casual conversation in the kitchen that is now missing.</p>
<p>Working out how not to lose the social capital we’ve built up in the office, and building it up with new starters is something we all need to address. For a start, things like social events (in work time) are a big part of that, and I make sure to attend as often as I can, both to try to make sure I don’t become a distant, completely unapproachable figure, and also because they are often extremely fun.</p>
<p>*If you are on my team and want one of these, even if just for a chat, Slack me.</p>
The Multiple Faces of Technical Leadership2020-07-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/tech-leadership-panel.html<p>On Monday I was on a panel, organised by Codemotion, about technical leadership. It was a great discussion, with extremely impressive co-panellists <a href="https://twitter.com/omosolatweets">Omosola Odetunde</a> and <a href="https://twitter.com/patkua">Pat Kua</a>, and well faciliated by <a href="https://twitter.com/pigiuz">Piergiorgio Niero</a>.</p>
<p>We discussed three main areas:</p>
<ol>
<li>What challenged each of us the most as a senior technology leader at different stages of our growth.</li>
<li>What we delegate, and what are the key responsibilities we keep on ourselves.</li>
<li>What roles we see in senior technical leadership.</li>
</ol>
<p>It was a really interesting discussion. You can watch it here:</p>
<div class="embedded">
<iframe src="https://www.youtube-nocookie.com/embed/gWXYVOjb-tA" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
Unlocking value with durable teams2020-06-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/durable-teams.html<p>Over the past six months, my group at the Financial Times has moved from project-based teams to durable teams. In this post I’ll explain why we made that move and how this is helping us deliver bigger and better things for our customers.</p>
<h2 id="we-want-to-be-able-to-deliver-as-much-value-as-possible">We want to be able to deliver as much value as possible</h2>
<p>Our goal on Customer Products is to make FT.com and the iOS and Android apps be as good as they can be for our customers, and increase the lifetime value of each customer or corporate licence. As with any tech organisation, we have a tech estate that we need to keep <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">sustainable, supportable and operationally reliable</a>, while also freeing up as much time as possible to deliver value to our customers and the business.</p>
<p>We use everything we already know about good software development to allow us to focus on where we can really add value. We already do continuous integration, automated testing, effective code review, and many more standard development practices that eliminate wasted time and low value work, and we also experiment, release early and iterate so we don’t spend too much time on work before we know it’s valuable.</p>
<p>However, we realised there was something we could do around the actual organisation of the group itself to help us become a really high-performing team.</p>
<h2 id="we-used-to-have-initiative-based-teams">We used to have initiative-based teams</h2>
<p>When I first started working at the Financial Times, my group — which builds the ft.com website and the iOS and Android apps — was made up of around 11 teams. Teams were formed around projects, usually with a product focus. For example, we had a Conversion team, whose focus was on converting casual readers into subscribers. When we had new initiatives, for example launching FT podcasts, we formed a new team.</p>
<p>Over the past year, we’ve changed to a new team structure: durable teams. This means our group is made up of nine teams, that each own a product area and the associated tech estate, plus our <a href="https://medium.com/ft-product-technology/why-and-how-we-changed-the-way-we-support-ft-com-7572371f3f36">ops cops team</a>, who handle in-hours support.</p>
<p><img src="/img/nine_teams.png" alt="A diagram showing the nine durable teams and how they overlap, along with our small ops cops team" /></p>
<h2 id="initiative-based-teams-have-some-advantages">Initiative-based teams have some advantages</h2>
<p>The main advantage of our previous structure is that each team is focused on one product goal, which is good from the perspective of working together to deliver value.</p>
<p>It also allows the group to quickly swarm on problems, and by not having to care about legacy code or bugs, the team are free to work as fast as they can on new features. It was a good approach for the early stages of building the new FT.com, where there was no legacy code to worry about.</p>
<p>It also makes budgeting easy — if we get investment to do an initiative, then we put a team together to do that initiative.</p>
<p>However, there are downsides to this approach, and over time they get worse.</p>
<h2 id="forming-storming-norming-perf-">Forming, storming, norming, perf-</h2>
<p>One big disadvantage of changing teams quite frequently, is that it often does not give the team a chance to form into a really high performing unit.</p>
<p>All teams go through a process of team building. One way to represent the stages a team goes through is <a href="https://hr.mit.edu/learning-topics/teams/articles/stages-development">forming, storming, norming, performing</a>, as they figure out how to work together in order to achieve the best outcomes.</p>
<p>If you build teams around projects, this means that there is a ramp-up period while the team go through the early stages, and if the team is not together for long enough, it’s possible they won’t spend very long (if any time at all) in the performing stage. This happens if new initiatives come in and the team is disbanded to form other, new teams.</p>
<p>This is frustrating for the people on the team, and also means the business does not get the best value out of the highly skilled people working for it.</p>
<h2 id="orphaned-technology-is-an-operational-risk-that-you-dont-know-about-yet">Orphaned technology is an operational risk that you don’t know about yet</h2>
<p>If the team is disbanded to move onto other projects, the technology they have built often does not move with members of the team to the new project. For example when the podcasts team disbands to work on other initiatives, none of the work they’ve done may fit into the new teams.</p>
<p>This is a problem waiting to happen; it is a security and operational risk. Without ownership of tech, if any future development or maintenance is needed, it may be hard to find someone who has time to do it. In some cases, everyone who worked on a project may have left the company.</p>
<p>Even if no security risks arise, it slows things down. If an unowned piece of tech has outlived its usefulness, there won’t be anyone to take care of decommissioning it, meaning that over time your tech stack gets clogged up with code that is not adding value but is adding complexity and slowing everything down.</p>
<h2 id="project-based-teams-remove-agency-from-the-team-and-add-administrative-cost">Project-based teams remove agency from the team (and add administrative cost)</h2>
<p>It takes a lot of effort to form a team; you need to find the right balance of skills, someone to tech lead, delivery manage and be the product owner; you need to find engineers, customer researchers, product designers and possibly business analysts and you need to make sure that the work they are currently doing can be finished or stopped.</p>
<p>It also means the initiative ideas are coming from outside the team. The idea for the project comes about before the team is formed, so by the time the team has formed, it’s already been decided what they are working on, even though that might not represent the biggest opportunity for the business or <a href="https://svpg.com/four-big-risks/">even be a feasible idea</a>.</p>
<p>When you bring a team together to deliver a project that has already been decided, this can lead to the team feeling like a <a href="https://medium.com/hackernoon/12-signs-youre-working-in-a-feature-factory-44a5b938d6a2">feature factory</a>, which is demotivating, and also doesn’t allow people to use their best skills to help solve business or customer problems.</p>
<h2 id="we-decided-to-move-to-durable-teams">We decided to move to durable teams</h2>
<p>While we were some way from being a feature factory (for example, we have a really strong culture around <a href="https://vimeo.com/273987309">measuring the outcomes of our work</a>, and <a href="https://2018.continuouslifecycle.london/sessions/how-we-ab-test-at-the-financial-times-and-what-happened-when-we-started-testing-headlines-too/">A/B testing features</a> to see if they are useful) some of the issues highlighted in the feature factory blog post were a little close to the bone, for example, “Team Tetris”.</p>
<p>We realised that one way to address a lot of the issues we had was to move to a durable team structure. This would allow us not just to solve some of the problems we currently had, but also it would allow us to really grow and unlock the potential of our teams.</p>
<ul>
<li>We can create teams that between them own our entire estate. Each team can take responsibility for the strategic direction of that area and kill things that no longer add value. We will no longer have unowned tech, which leads to security and operational risk.</li>
<li>Durable teams mean we do not waste effort stopping and starting teams every quarter.</li>
<li>Durable teams help people build domain knowledge, mastery and vision, help build organisational memory and fine tune effective ways of working.</li>
<li>When a team reaches peak productivity, ideas and execution are consistently better. From meaningful work we get motivated people and a better quality product.</li>
<li>Durable teams help us be clear about what our strategy is, what is in scope and what our capacity is.</li>
<li>Durable teams make it easier for those outside our group to know who to talk to about a particular part of the product.</li>
</ul>
<h2 id="a-long-lived-team-can-work-on-bigger-opportunities">A long-lived team can work on bigger opportunities</h2>
<p>A durable team is in a good position to think strategically about the part of the domain they own. When the team knows they are going to be around to experiment, iterate and see the outcomes from changes they make, then they can take bigger bets.</p>
<p>This works from both a product and a technical perspective. A long lived team can take a big bet on changing how we show you relevant news because they will be around to make the changes and keep iterating until they get it right. They can also take a big bet on moving database platform to unlock potential, because they will still be around to reap the benefits of the move.</p>
<p>When a team is just there for a short, or uncertain, length of time, then in order to make any impact, they have to stick to small, incremental improvements. Working on the big bets which might reap huge rewards is too risky — the team may be disbanded before they’ve had a chance to show the benefit. If they have a longer lifespan they are free to try some things that are higher risk. Some will fail, but some will lead to the big changes that lead to exponential growth.</p>
<h2 id="stable-teams-can-be-empowered-teams">Stable teams can be empowered teams</h2>
<p>What we really want is <a href="https://svpg.com/empowered-product-teams/">empowered teams</a>, coming up with and trying out the ideas that can really deliver value to our customers and help the business grow.</p>
<p>Each of our durable teams has a tech lead, a product manager and a delivery manager. Together, they are responsible for setting the direction for the team, working out what opportunities to focus on, and weighing up priorities within the team. This includes when to focus on adding product features and when it’s more crucial to focus on improving the tech.</p>
<p>As a leadership team, we can step back from making sure we know the detail of exactly what each team will do and concentrate on setting the higher level strategy and objectives. Then we can trust the team to use their huge skills and experience in the domain to identify for themselves what is the most valuable work they could be doing.</p>
<h2 id="technical-ownership-of-our-estate-has-already-started-to-save-the-business-money">Technical ownership of our estate has already started to save the business money</h2>
<p>Our tech strategy on Customer Products is <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">No next Next</a>. We don’t want our tech to drift so far off track we have to spend £10m and another two years rebuilding it. Making sure all our tech is owned is part of that strategy. When all tech is owned by a currently active team, the team that owns that part of the estate can make longer-term decisions about it.</p>
<p>When we started the move to durable teams, we had 332 systems or repositories, of which 272 were not assigned to a specific team. This didn’t mean that no-one knew about any of them, but we didn’t know which ones people knew about and which were orphaned — we didn’t know the scale of the problem.</p>
<p>Now, all of our systems or packages are owned by a currently active team. As teams started taking ownership, they quickly identified systems that were no longer needed or no longer used. By the time we finished tech ownership, we had removed 47 systems and packages, and more consolidation is planned.</p>
<p>This saves the business money directly in ongoing infrastructure costs, and indirectly, both through engineers no longer needing to maintain these systems, and through making <a href="https://medium.com/ft-product-technology/why-and-how-we-changed-the-way-we-support-ft-com-7572371f3f36https://medium.com/ft-product-technology/why-and-how-we-changed-the-way-we-support-ft-com-7572371f3f36">operational support easier</a>.</p>
<h2 id="handling-work-that-falls-across-teams">Handling work that falls across teams</h2>
<p>There are two main risks in having a durable team set up like this. One is that some work naturally falls across two or more teams. Previously we would have formed a team to work on that thing, but now we need to coordinate between the existing teams to get that done.</p>
<p>The way we address that is as well as Delivery Managers co-ordinating as they usually do, the Principal Engineers — who work across the whole of Customer Products — make sure they have technical oversight and input into those projects. In addition, as far as possible, we try to architect the work so it can be done asynchronously, perhaps released behind a feature flag, and linked together when work in all teams is complete.</p>
<h2 id="making-sure-splitting-the-domain-doesnt-limit-opportunities">Making sure splitting the domain doesn’t limit opportunities</h2>
<p>The other risk is that splitting the domain up into set teams has the potential for limiting thinking. Where you might have previously looked for the biggest opportunity across the whole estate, you might now only look for the biggest opportunity in your domain.</p>
<p>The current situation is still better than it was, in that there is much more potential for teams to make those strategic decisions even within their own domain, as they own everything around that domain. But it is an important part of my leadership role, as Tech Director, alongside the Product Director and Delivery Director, to make sure that this opportunity for bigger picture thinking is not lost.</p>
<p>It’s also worth noting that the current durable teams are not set in stone. It may be that as the product strategy develops over the coming years the actual teams change. However, we will approach any future changes with the same principles: empowerment, total tech ownership and allowing teams to stay together for the long term.</p>
<h2 id="we-got-it-done-just-in-time">We got it done just in time</h2>
<p>We had the last handover durable team meeting on the 24th February and the FT started working from home on Monday 16th March.</p>
<p>We are so lucky that we got the durable teams work finished in time. Having a durable team is really valuable at a time like this. It would be extremely difficult to manage if we were still trying to put teams together to work on initiatives, while all working from home in difficult and stressful circumstances, and those teams had to bond, figure out how to work together and perform in these circumstances.</p>
<h2 id="its-already-adding-value">It’s already adding value</h2>
<p>We’ve already seen how durable teams helps us go faster. We’ve eliminated some waste in our technical estate and teams are looking for ways to reduce it further. We recently had to take on a piece of work that we hadn’t planned for, and we were able to distribute it among the relevant teams and get started on it, without waiting to hire people for the extra work, and we know when it’s finished it will continue to be owned by those teams. And we’ve seen great ideas emerging from teams, including some excellent discovery work that’s leading to teams taking much bigger bets.</p>
<p>Like many others the FT has been <a href="https://www.ft.com/content/0069d422-da44-4d06-ae5b-766836bfda6e">hit by the coronavirus crisis</a>. Having durable teams and a strategy of a <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">sustainable, flexible codebase</a> gives us the foundation to accelerate fast and have greater flexibility to support whatever the business needs, through this crisis and beyond.</p>
<p><em>This post originally appeared on the <a href="https://medium.com/ft-product-technology/unlocking-value-with-durable-teams-a70efb435a19">FT Product & Technology blog</a></em>.</p>
Why and how to do annual performance reviews2020-06-23T00:00:00+00:00https://www.annashipman.co.uk/jfdi/end-year-reviews.html<p>We are coming up to mid-year review time in my group at the FT, and two brilliant engineers on my team, <a href="https://twitter.com/GlynnPhillips">Glynn Phillips</a> and Charlotte Payne, decided to run a session for the other line managers in our group on why we have this process.</p>
<p>Glynn said they were thinking about it at a broader level: “It could be easy for us to answer the “why do we do this” question and it still sounds to some people like ‘Well it’s an FT procedure’.”</p>
<p>They asked for my views, and it turned out that I had a lot to say on this topic. Here is what I replied. [Note: these are my views and not a formal statement of the FT’s views!]</p>
<h2 id="end-of-year-reviews--why-do-we-do-them">End of year reviews – why do we do them?</h2>
<p>The end of year review is an opportunity to reflect on your progress throughout the year.</p>
<p>Ideally you, with your manager, will have set yourself some objectives at the beginning of the year, but even if you have not, it’s an opportunity to reflect on the past year, feel good about things you’ve done well and think about areas where you could have done things better.</p>
<p>In an ideal world we would reflect on our work all the time (that’s why we have retrospectives, for example), but it’s easy for that to fall by the wayside. The end of year review is the one time where, because we have a process that is formalised by the department, we have to make sure we make time for that.</p>
<p>If you’re not interested in being better at your job or furthering your career, the end of year review can be a tedious formality, but if you are, it’s a great opportunity to make sure you carve out time to reflect and plan ahead.</p>
<h2 id="its-a-formalised-time-to-get-feedback">It’s a formalised time to get feedback</h2>
<p>The end of year review is an opportunity to get feedback from your manager, reports, peers and people in other groups/departments. [Note: in our department at the FT, we do 360-degree feedback using a unified form twice a year, and people are encouraged to prioritise it at this time].</p>
<p>Again, ideally the feedback loop would be constant; you would always be giving and seeking feedback throughout the working week, but in case this hasn’t happened as often as would be ideal, the end of year review is a formal time to make sure this happens.</p>
<p>We have a great process in Product and Tech where there is an expectation that we give feedback via the unified feedback process at review time. While this can lead to being overloaded with feedback requests, the advantage is that for review time, it’s expected that people prioritise it, so you will end up receiving feedback. If it wasn’t formalised twice a year, again, it would be easy for this to end up not happening.</p>
<p>It’s also useful getting a lot of feedback at once, so you can see what areas multiple people give you feedback on, vs. feedback that is an outlier.</p>
<p>Finally, with feedback, it is a gift, and you can always choose not to accept it. You do not have to act on every piece of feedback you receive.</p>
<h2 id="be-mindful-when-passing-on-feedback-as-a-manager">Be mindful when passing on feedback as a manager</h2>
<p>As a manager, it’s important to review the feedback and make sure you are thinking about biases when you go through the feedback with your report.</p>
<p>For example, <a href="https://web.stanford.edu/dept/radiology/cgi-bin/raddiversity/wp-content/uploads/2017/12/TheAbrasivenessTrap.pdf">“Abrasive” is a word that is only applied to women</a>; for men the feedback might say approvingly that he was direct, but more likely it wouldn’t be mentioned. <a href="https://www.theatlantic.com/business/archive/2015/10/why-black-workers-really-do-need-to-be-twice-as-good/409276/">Black workers tend to receive extra scrutiny from managers and more attention on mistakes</a> than white workers, so feedback that would be overlooked for a white person might be mentioned for a black person. And there are many, many other examples.</p>
<p>[Edit, 11/09/2020: adding <a href="https://biasinterrupters.org/wp-content/uploads/Identifying-Bias-in-Performance-Evaluations-Guide-no-citations.pdf">this extremely useful resource: Bias Interruptors</a>; examples of how bias can appear in performance reviews and steps to address them, shared with me by the excellent <a href="https://www.linkedin.com/in/keran-braich/">Keran Braich</a>.]</p>
<p>It’s also very important to make sure you ask a wide variety of people, not just the ones your report has suggested you ask. I usually ask my reports if there is anyone they’ve had conflict with to get feedback on; it may not be the most positive or constructive feedback, but it will often be very useful, even if you both choose to disregard it.</p>
<p>So while the feedback is very useful, it’s important as a manager you don’t take it as the final say, but as one of the factors that helps you form an overall picture, tell your report if you suspect they should take the feedback with a pinch of salt, and listen to them when they point out biases you may not have seen.</p>
<h2 id="we-give-a-performance-review-mark">We give a performance review mark</h2>
<p>For the FT, the output of the performance review is a mark, nominally on a scale of 1–5. Roughly speaking, 3 is where we should all be at. 3 means you are performing well against challenging objectives. 4 means that you are exceeding those objectives; it’s good to get a 4 but also is an indication that perhaps you are due for a promotion as your current objectives are not challenging enough. 5 means you are really knocking it out of the park; you are performing absolutely exceptionally. It’s rare to get a 5.</p>
<p>At the other end of the scale, a 2 means you are not meeting your objectives, and if you get a 2 then you and your manager need to look at doing performance improvement, to get you up to a level where you are meeting your objectives. A 2 is not necessarily a bad thing; it can be an extremely useful wake-up call that your peformance isn’t aligned with what the organisation is expecting from you, and is a good opportunity to get the support you need to get back on track.</p>
<p>I don’t think 1 is a mark that is ever actually given to someone. If someone is in line to get a 1, then they probably should not be working here, or at least not in that role (so it’s not a normal scale).</p>
<p>For the FT the value of the mark is to see where there are areas that need attention. A 2 indicates to HR that a manager needs support to do performance improvement. A 4 or 5 year after year for the same person indicates they are in the wrong job. A team where every member is getting a 5 indicates something that needs closer attention.</p>
<p>It’s a blunt instrument, but understandable for an organisation of > 1,000 people.</p>
<p>It can support conversations with your manager as well. “What would I need to do to get a 4 this year?” is a perfectly reasonable question and can help focus the mind.</p>
<h2 id="the-mid-year-review--why-do-we-do-that">The mid-year review – why do we do that?</h2>
<p>Because the end of year review process results in a mark, the mid-year review process is an opportunity to check where you are heading. Nothing in an end of year review should come as a surprise to you – if there are issues or great performance, your manager should already have talked to you about those things – but the mid-year review is a formal point at which to talk to your manager about your trajectory, and review your objectives to see if you are actually half-way along with them at this point or if, instead, you need to change them.</p>
<p>Again, all these are things that in an ideal world you would always be thinking about and reflecting on, but in the day to day it’s very easy to forget to focus on the bigger picture of your own personal development.</p>
<h2 id="how-would-i-gauge-whether-someone-is-performing-at-a-good-standard">How would I gauge whether someone is performing at a good standard?</h2>
<p>Giving someone a mark is a combination of how they are performing against their objectives, how they are doing relative to their level in the career competencies framework, the feedback you receive, and anything else you know about their work (for example, you may have seen them present at Snapshot [Note: our monthly team meeting], or seen some good pull requests, or thoughtful pull request reviews, or seen them asking good question in the dev huddle, etc).</p>
<p>It can be hard to know how someone is doing if you don’t work with them day to day (for example if they are in a different team) but when you put together all the sources of information you have, you can build up a good picture.</p>
<h2 id="they-are-really-worth-doing">They are really worth doing</h2>
<p>My view is that the mid-year review and end of year review are really useful; even though it can involve a lot of admin from both manager and report, it’s a great opportunity to make time for the important work of your own career development and to help your colleagues with theirs.</p>
A 'Hello World' mod-wsgi application2020-06-06T00:00:00+00:00https://www.annashipman.co.uk/jfdi/mod-wsgi-hello-world.html<p>I recently found myself having to use <a href="https://modwsgi.readthedocs.io/en/develop/index.html"><code class="language-plaintext highlighter-rouge">mod_wsgi</code></a>, an Apache module which can host Python web apps that support the WSGI spec.</p>
<p>The <a href="https://modwsgi.readthedocs.io/en/develop/getting-started.html">docs</a> say that you shouldn’t attempt to use an app dependent on a framework like Django until you’ve got a basic ‘Hello World’ app running first, which will validate that “you at least understand the basics of configuring Apache.”</p>
<p>That’s kind of patronising; I do <em>at least</em> understand the basics of configuring Apache, but it seems this “simple to use” (their words, of course) module needs a bit more explanation.</p>
<p>Unfortunately, their <a href="https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html">“Quick configuration guide”</a> was anything but, so here’s a better one.</p>
<h2 id="starting-from-scratch-using-ubuntu-1804">Starting from scratch, using Ubuntu 18.04</h2>
<p>Install Apache:</p>
<p><code class="language-plaintext highlighter-rouge">sudo apt-get install apache2</code></p>
<p>(If you are using this in anger you probably want to install some other things here too, e.g. <code class="language-plaintext highlighter-rouge">apache2-utils</code> and <code class="language-plaintext highlighter-rouge">ssl-cert</code> but for the purposes of this basic app, not required.)</p>
<p>Install the Python3 version of <code class="language-plaintext highlighter-rouge">mod-wsgi</code>:</p>
<p><code class="language-plaintext highlighter-rouge">sudo apt-get install libapache2-mod-wsgi-py3</code></p>
<h2 id="create-the-basic-hello-world-wsgi-app">Create the basic Hello World WSGI app</h2>
<p>Create a file called <code class="language-plaintext highlighter-rouge">myapp.wsgi</code>. Ideally not in your document root so you don’t leak the source code. I put it in <code class="language-plaintext highlighter-rouge">/home/anna/code/myapp.wsgi</code>.</p>
<p>Here’s the content of the file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def application(environ, start_response):
status = '200 OK'
output = b'Hello World!'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
</code></pre></div></div>
<p>Note the <code class="language-plaintext highlighter-rouge">b</code> on line 3, the <a href="https://stackoverflow.com/questions/34838443/typeerror-sequence-of-byte-string-values-expected-value-of-type-str-found">output needs to be bytes</a>.</p>
<h2 id="tell-mod_wsgi-where-to-look-for-code">Tell <code class="language-plaintext highlighter-rouge">mod_wsgi</code> where to look for code</h2>
<p>Create a file called <code class="language-plaintext highlighter-rouge">mod-wsgi.conf</code> in <code class="language-plaintext highlighter-rouge">conf-available</code>:</p>
<p><code class="language-plaintext highlighter-rouge">/etc/apache2/conf-available/mod-wsgi.conf</code></p>
<p>In the file, set the <code class="language-plaintext highlighter-rouge">WSGIScriptAlias</code>: the URL path (<code class="language-plaintext highlighter-rouge">/</code> if you want it at the root) and the location of the above WSGI app on your file system.</p>
<p><code class="language-plaintext highlighter-rouge">WSGIScriptAlias /myapp /home/anna/code/myapp.wsgi</code></p>
<p>You also need to give permission to accces that directory if it’s not in your document root. So the full file is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WSGIScriptAlias /myapp /home/anna/code/myapp.wsgi
<Directory /home/anna/code>
Require all granted
</Directory>
</code></pre></div></div>
<p>Enable the configuration:</p>
<p><code class="language-plaintext highlighter-rouge">sudo a2enconf mod-wsgi</code></p>
<p>And then reload Apache for the config to take effect.</p>
<p><code class="language-plaintext highlighter-rouge">sudo systemctl reload apache2</code></p>
<p>You could also put this config into <code class="language-plaintext highlighter-rouge">httpd.conf</code> instead of into the <code class="language-plaintext highlighter-rouge">conf-available</code> directory.</p>
<h2 id="you-can-now-see-hello-world">You can now see “Hello World”</h2>
<p>At <code class="language-plaintext highlighter-rouge">http://SERVER_IP/myapp</code></p>
<h2 id="basic-app-is-done-but-there-are-a-couple-of-other-considerations">Basic app is done, but there are a couple of other considerations</h2>
<p>There are a few things you’d want to do differently if doing this for real. For starters, this is server scope, but you probably want to put it in the VirtualHost for your site. It’s also recommended that you run it in daemon mode.</p>
<h2 id="scoping-it-to-your-domain">Scoping it to your domain</h2>
<p>In <code class="language-plaintext highlighter-rouge">/etc/apache2/sites-available</code> create a file to put the virtual host in. I created <code class="language-plaintext highlighter-rouge">/etc/apache2/sites-available/mydomain.conf</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerAlias www.mydomain
DocumentRoot /var/www/mydomain
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
</code></pre></div></div>
<p>Enable that config:</p>
<p><code class="language-plaintext highlighter-rouge">sudo a2ensite mydomain.conf</code></p>
<p>and then reload Apache:</p>
<p><code class="language-plaintext highlighter-rouge">sudo systemctl reload apache2</code></p>
<p>To scope the <code class="language-plaintext highlighter-rouge">mod_wsgi</code> configuration to your domain, put it in that virtual host, i.e.:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerAlias www.mydomain
DocumentRoot /var/www/mydomain
WSGIScriptAlias /myapp /home/anna/code/myapp.wsgi
<Directory /home/anna/code>
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
</code></pre></div></div>
<h2 id="setting-wsgipythonhome">Setting WSGIPythonHome</h2>
<p>You might find this doesn’t work because Apache can’t find Python. In that case you need to set the WSGIPythonHome. This should be the path to your virtualenv.</p>
<p><code class="language-plaintext highlighter-rouge">WSGIPythonHome /home/anna/code/myproject/env</code></p>
<p>You <a href="https://serverfault.com/a/235624">can’t set the WSGIPythonHome inside the Virtual Host</a>, so put the above line into <code class="language-plaintext highlighter-rouge">httpd.conf</code>.</p>
<p>As above, you could also put the whole lot into <code class="language-plaintext highlighter-rouge">httpd.conf</code> instead of into the <code class="language-plaintext highlighter-rouge">sites-available</code> diretory, but if you do that, make sure the <code class="language-plaintext highlighter-rouge">WSGIPythonHome</code> line is outside of the Virtual Host block.</p>
<p>To find the virtualenv home, activate the virtualenv, start a python shell, and then:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>>import sys
>>>sys.prefix
</code></pre></div></div>
<h2 id="running-it-in-daemon-mode">Running it in daemon mode</h2>
<p>It’s recommended that you run it in daemon mode, because otherwise it is in ‘embedded mode’ which requires a restart to Apache whenever you change the <code class="language-plaintext highlighter-rouge">mod_wsgi</code> configuration.</p>
<p>For that, you need to set <code class="language-plaintext highlighter-rouge">WSGIDaemonProcess</code> and <code class="language-plaintext highlighter-rouge">WSGIProcessGroup</code>. Here is what I did, but you can <a href="https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process">read more on the <code class="language-plaintext highlighter-rouge">mod_wsgi</code> docs</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WSGIDaemonProcess myproject python-path=/home/anna/code/myproject processes=2 threads=15
WSGIProcessGroup myproject
</code></pre></div></div>
<p>So the complete VirtualHost is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerAlias www.mydomain
DocumentRoot /var/www/mydomain
WSGIScriptAlias /myapp /home/anna/code/myapp.wsgi
WSGIDaemonProcess myproject python-path=/home/anna/code/myproject processes=2 threads=15
WSGIProcessGroup myproject
<Directory /home/anna/code>
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
</code></pre></div></div>
<p>This works for my Django project.</p>
<h2 id="why-did-that-all-work">Why did that all work?</h2>
<p>Armed with the instructions on how to get a basic ‘Hello World’ app running, it might now be helpful to go back and read the erroneously named <a href="https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html">Quick configuration guide</a>. You have to fill in some dots yourself, so this is what this post is for.</p>
The difficult teenage years at Tech Nottingham2020-05-13T00:00:00+00:00https://www.annashipman.co.uk/jfdi/tech-nottingham.html<p>I was meant to go to Nottingham this week to give a talk about tech strategy. It was going to be a follow on from my talk at Continuous Lifecycle, <a href="/jfdi/after-the-launch.html">After the Launch: the difficult teenage years</a> – where are we a year later and what have I learned?</p>
<p>In the circumstances, travel to Nottingham was out, but the brilliant folk at Tech Nottingham <a href="https://www.technottingham.com/events/tech-nottingham-may-2020">have gone online</a>, and I was able to give the talk in a slightly different format than planned, from the comfort of my own home!</p>
<p><img class="half-width" src="/img/tech_nottingham_screenshot.jpg" alt="Screenshot of some Tech Nottingham participants on a Zoom call" /></p>
<p>Instead of the planned new talk, we watched the video of the previous talk, and then we did a Q&A. A year on, it all holds true, we still have the same tech strategy and it’s delivered some brilliant results, so the previous talk was still relevant.</p>
<p>It was great to revisit it (though awkward to watch myself – I turned my camera off so the other Zoom participants couldn’t see me cringing at my errors!) and there were some excellent questions.</p>
<p>The Tech Nottingham folk have put a lot of work into making the online event work, with ice-breakers, open spaces and a competition (people could win copies of <a href="/jfdi/good-strategy-bad-strategy.html">Good Strategy, Bad Strategy</a>). Everyone is welcome at <a href="https://www.technottingham.com/">Tech Nottingham</a>, and now, you don’t even have to live in Nottingham to attend!</p>
Social distancing quiz2020-03-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/social-distancing-quiz.html<p>With our whole team suddenly working from home, we’ve put in a regular Friday 3pm “remote social” for the whole of Customer Products, a team of about 60 people. The idea is to help with the loss of the casual chat and social interaction we get in the office. Last Friday, for the first one, I ran a remote quiz.</p>
<h2 id="the-technicalities">The technicalities</h2>
<p>We did it on Google Meet, (as a meeting rather than a live stream) and there were 38 people on the hangout. Prior to the quiz I’d asked everyone to self-organise into teams using a Google spreadsheet.</p>
<p>I then asked everyone to set up another channel with their team – a Slack group, WhatsApp group or another Hangout (I didn’t see anyone speaking so I assume it was for most people one of the former).</p>
<p>It wasn’t set up in any way to make it hard to Google answers, which is obviously much easier if you’re already sat at the computer, so I just said “don’t cheat!”.</p>
<p>My initial idea for marking was everyone put their answers into a Google doc and then between rounds, share with another team for marking, but this was chaos; after I’d gone through the answers several teams realised no-one had marked theirs. Then someone pointed out that if we were trusting each other not to cheat, we could also trust each other to mark our own answers, so that’s what we did after that. They then all Slacked me their scores and I put them into a spreadsheet.</p>
<p>One other thing which added to the sense of connectedness, was that as I was going through the answers, people called theirs out, which helped make it feel less one-way.</p>
<h2 id="the-rounds">The rounds</h2>
<p>We had two music rounds, two picture rounds, a currencies & podcasts round and a film quotes round. The questions and picture rounds were related to our work at the <a href="https://www.ft.com/">Financial Times</a>.</p>
<p>The first music round was film songs, prepared for a previous quiz by <a href="https://twitter.com/nickramsbottom">Nick Ramsbottom</a>. This was a WhatsApp Audio mpeg, and I just played this on my computer.</p>
<p>Then, after I’d given the answers to that, while I was collecting in scores, I shared a link to the first picture round, which was “Name the FT journalist”, created by <a href="http://ft.com/luke">Luke Kavanagh</a>.</p>
<p>The third round was currencies and podcasts (feel free to have a go at this one yourself, I’ve copied it below!).</p>
<p>I then did a music intros round. This was something I’d prepared for a quiz for some friends about 12 years ago, and this kept going wrong; I had all the files in a folder as <code class="language-plaintext highlighter-rouge">.m4a</code>s, but if I didn’t stop them quickly enough, iTunes just segued randomly into something else in my iTunes libary, including intros that I’d cut in the interests of time and other tunes completely unrelated (was it shuffling through my entire library? I do not know and did not have time on the spot to find out). The lesson there is: be like Nick and put it all into one file in advance!</p>
<p>While collecting scores for those three rounds, I distributed the next picture round, which was based on <a href="https://www.ft.com/columnists/person-in-the-news">‘People in the news’</a>, a weekly FT column about someone who is in the news at that time, also put together by Luke Kavanagh.</p>
<p>Finally, the last round was film quotes. 1 point for film, 1 for actor and a bonus point for character name. Two of my favourites: “I remember every detail. The Germans wore grey, you wore blue.” and “I guess I picked the wrong week to quit smoking”.</p>
<p>The latter is a favourite because there are in fact 6 available points for that question; the film you probably know it from (and actor and character name), and then the completely straight non-parody film that the first one is based on. (If you’re looking for lockdown activity, watching the two consecutively is fun).</p>
<h2 id="currencies-and-podcasts">Currencies and podcasts</h2>
<p>Here’s a round if you want some light entertainment.</p>
<p>What are the currencies of the following countries?</p>
<ol>
<li>Thailand</li>
<li>Israel</li>
<li>Iraq</li>
<li>Hungary</li>
<li>Kenya</li>
<li>Vatican City</li>
<li>Pakistan</li>
<li>Syria</li>
<li>South Africa</li>
<li>Peru</li>
</ol>
<p>Name the <a href="https://www.ft.com/podcasts">FT podcast</a> by filling in the blanks:</p>
<ol>
<li>C…… C…</li>
<li>T.. R…… R…..</li>
<li>B….. t.. M….</li>
<li>News B….ing</li>
<li>M…. S…</li>
</ol>
<h2 id="what-would-i-do-differently-next-time">What would I do differently next time?</h2>
<p>The quiz overran – we had an hour slot but it actually went to two. I hadn’t realised how long it would all take. Once we realised, most were happy to have it overrun, but if I’d realised in advance, I could have split it in two and had two quizzes for the price of one. My feeling is it is worth trying to keep future socials to little and often because it can get a bit tiring.</p>
<p>A couple of teams missed rounds where I shared the link in the chat. I should have been extra clear on instructions, because when everyone is distributed you don’t pick clues up from what other teams are working on.</p>
<p>Other than that, it went pretty well, I think!</p>
<h2 id="i-would-definitely-do-it-again">I would definitely do it again</h2>
<p>It was a lot of fun – at least, I had fun, and other people seemed to as well. Someone said “THANK YOU so much for doing this quiz, it’s just what I needed, it’s really helping me feel connected with everyone and was great fun!” and my favourite comment in Slack was “this is the most amazingly confusing and wonderful pub quiz I’ve ever taken part in. I have literally no idea what is going on and I’m still enjoying it”.</p>
<p>I think it really helped to have a social event and it was nice to see everyone (and some kids!).</p>
<p>Same time next week. I need to work out how to do charades such that the person silently doing the charades is the main focus, rather than the people shouting out answers. A backing soundtrack? Any suggestions, let me know <a href="https://twitter.com/annashipman">on Twitter</a>!</p>
Slack2019-12-31T00:00:00+00:00https://www.annashipman.co.uk/jfdi/slack.html<p>I’ve written briefly about <a href="https://www.amazon.co.uk/Slack-Getting-Burnout-Busywork-Efficiency-ebook/dp/B004SOVC2Y">Slack</a> before (at the end of <a href="/jfdi/high-output-management.html">my post on High Output Management</a>), but this year I decided to re-read it. It’s good. Here are my notes from the second read.</p>
<h2 id="the-idea-of-maximum-efficiency-is-a-myth">The idea of maximum efficiency is a myth</h2>
<p>Using the analogy of a <a href="http://www.jeanierhoades.com/wp-content/uploads/2012/01/old-slide-puzzle.jpg">sliding tile puzzle</a> he points out that if you fill the empty slot, it’s a more more efficient use of space, but then it’s impossible to change. That’s why you need slack at work; if your time is already filled to capacity you will not be able to deal with emerging issues, or (as <a href="/jfdi/effective-executive.html">Drucker points out</a>, plan for the future).</p>
<p>By seeming to thriving under pressure you put pressure on your peer managers.</p>
<h2 id="leadership-and-strategy">Leadership and strategy</h2>
<p>When an organisation is in drift and a leader takes the helm, there is relief.</p>
<p>Leadership is encouraging people to take short term pain for long term gain, because we all tend to be short term thinkers. Giving trust is a way to get trust.</p>
<p>However, he is not a fan of <a href="https://en.wikipedia.org/wiki/OKR">OKRs</a>: he says management by objectives guides you to make tactical adjustments only, under the limitation that you don’t mess with current strategy, whatever it might be.</p>
<h2 id="you-need-to-make-explicit-provision-for-failing-safely">You need to make explicit provision for failing safely</h2>
<p>Being safe to fail means making explicit provision for lots of small but expensive failures along the way to big success.</p>
<p>You can’t count on all your risks paying off; you need to set aside time and money. The science of risk management guides you as to how much.</p>
<h2 id="risk-management-is-probabilistic">Risk management is probabilistic</h2>
<p>The explicit declaration of uncertainties allow you to manage a sensible risk reserve across the whole portfolio. Organisations can’t be aggressive about risk-taking without meaningful assessment of the extent of the uncertainties.</p>
<p>Without sensible risk management, organisations become risk averse.</p>
<h2 id="how-to-maintain-a-risk-register">How to maintain a risk register</h2>
<ol>
<li>List each risk</li>
<li>Have an ongoing process for discovering new risks</li>
<li>Quantify each one as to potential impact and likelihood</li>
<li>Designate a transition indicator for each one that will tell you if the risk is beginning to materialise</li>
<li>Set down in advance what the mitigation plan is for each risk</li>
</ol>
<h2 id="set-aside-a-risk-reserve-of-cash">Set aside a risk reserve of cash</h2>
<p>This should be such that you have at least a 50/50 chance you’ll have enough to cover the risks that do materialise.</p>
<p>You may have to undertake activity that you won’t need if risks don’t materialise in order to be prepared for if they do.</p>
<h2 id="stop-doing-risk-free-stuff">Stop doing risk-free stuff</h2>
<p>You need the resources for transformational work, and you’ll never get them if you keep on doing non-transformational stuff.</p>
<h2 id="you-need-slack-to-be-able-to-get-things-done">You need slack to be able to get things done</h2>
<p>The difference between the time it takes you to arrive at all prudent speed and breakneck speed is slack. Slack is what helps you arrive quickly but with an unbroken neck.</p>
<h2 id="but-how-do-you-get-more-slack">But how do you get more slack?</h2>
<p>I enjoyed this book, and it’s a quick read with some good ideas. However, while it’s a good explanation of why you need slack, it’s light on actual suggestions for how to arrange your life to get it.</p>
<p>One resource I’ve found useful is this exercise, <a href="https://hbr.org/web/2013/08/assessment/make-time-for-work-that-matters">make time for the work that matters</a>, but for practical suggestions for how to manage your working life with rigour and direction, my go-to book is still <a href="/jfdi/high-output-management.html">High Output Management</a>.</p>
The Effective Executive2019-12-30T00:00:00+00:00https://www.annashipman.co.uk/jfdi/effective-executive.html<p>This year, I finally read <a href="https://www.goodreads.com/book/show/48019.The_Effective_Executive">The Effective Executive</a>. It was great. Here are some of the things I took from it.</p>
<h2 id="you-want-to-be-effective-rather-than-efficient">You want to be effective, rather than efficient</h2>
<p>Knowledge work is not defined by cost, it’s defined by results, and the effective executive has the ability to get the right things done.</p>
<h2 id="knowing-and-analysing-how-you-spend-your-time-is-the-most-important-thing">Knowing and analysing how you spend your time is the most important thing</h2>
<p>“Nothing else distinguishes effective executives as much as their tender loving care of time.”</p>
<p>Innovation and change make inordinate time demands on the executive. All one can think and do in a short time is to think what one already knows and do as one has always done.</p>
<p>Research, to be productive, has to be the disorganised, the creator of a different future and the enemy of today.</p>
<p>However, he says in any executive job a large part of the time must be wasted. Rarely as much as 25% of one’s time is discretionary. So you should set deadlines for the discretionary work.</p>
<h2 id="first-things-first-and-second-things-not-at-all">First things first and second things not at all</h2>
<p>Effective executives do first things first, and they do one thing at a time.</p>
<p>A decision has to be made about which tasks deserve priority – the only question is who makes the decision, the executive or the pressures. (This reminds me of “schedule maintanance for your machines before they schedule it for you”).</p>
<h2 id="you-need-self-discipline-and-an-iron-determination-to-say-no">You need self-discipline and an iron determination to say no</h2>
<p>Keeping many balls in the air is a circus stunt but even the juggler only does it for ten minutes or so.</p>
<p>The secret of people who do many things is that they do one thing at a time, so as a result they need much less time in the end.</p>
<p>People who get nothing done work a lot harder. They underestimate how long everything will take, don’t expect the unexpected, and do several things at once, so if any one thing falls behind, the whole house of cards collapses.</p>
<p>Concentration – that is, the courage to impose on time and events your own decision about what matters and comes first – is the only hope of being the master of events and time, rather than at their mercy.</p>
<h2 id="plan-for-the-activities-of-tomorrow-not-yesterday">Plan for the activities of tomorrow not yesterday</h2>
<p>Effective executives periodically review their work programme and ask, if we didn’t already do this would we start now? And unless the answer is a resounding yes they drop the activity or curtail it. Pull out their resources and put them on the activity of tomorrow.</p>
<p>Every executive has to spend time, energy and ingenuity on patching up or bailing out the actions and decisions of yesterday, whether their own or their predecessors. This takes up more hours of the day than any other task.</p>
<p>The need to slough off the outworn old to make possible the productive new is universal.</p>
<h2 id="dont-sacrifice-the-future-on-the-altar-of-today">Don’t sacrifice the future on the altar of today</h2>
<p>It is more productive to convert an opportunity into results than to solve a problem – which only restores the equilibrium of yesterday.</p>
<p>Executive work is always postponable because it doesn’t try to solve yesterday’s crises but to make a difference tomorrow. One abandons what one postpones.</p>
<p>The pressures favour the crisis over the opportunity, what has happened over the future, the immediate and visible over the real, and the urgent over the relevant.</p>
<h2 id="decide-your-posteriorities">Decide your ‘posteriorities’</h2>
<p>As well as priorities you should set ‘posteriorities’ – things not to tackle – and stick to them. One rarely over-prunes.</p>
<p>Circumstances change and so do priories and posteriorities – sometimes as a result of doing your priority task. That is why you must focus on one task, and then when that is done, figure out the next.</p>
<h2 id="the-most-important-part-of-priorities-and-posteriorities-is-courage">The most important part of priorities and posteriorities is courage</h2>
<p>Setting a posteriority is unpleasant. Every posteriority is someone else’s priority. It’s much easier to just do a little bit of everything – but nothing gets done.</p>
<ul>
<li>Pick the future against the past</li>
<li>Focus on opportunities not problems</li>
<li>Choose your own direction rather than climbing on the bandwagon</li>
<li>Aim high. Aim for something that will make a difference, rather than something which is safe and easy to do</li>
</ul>
<h2 id="most-effective-decisions-are-distasteful">Most effective decisions are distasteful</h2>
<p>Executives are not paid for doing things they like to do, they are paid for getting the right things done.</p>
<p>Decisions require courage as much as judgment. Make sure you are not uneasy, but also do not delay. Act or do not act, but don’t hedge. (Or, as I prefer to think of it, “Do. Or do not. There is no try.”)</p>
<p>One always has to start out with what is right rather than what is acceptable, because one always has to compromise in the end.</p>
<p>It is fruitless and a waste of time to worry about what is acceptable and what to say – the things one worries about never happen. And in the process of starting out with what is acceptable you lose any chance of coming up with the right answer.</p>
<h2 id="establish-a-principle-rather-than-making-decisions-case-by-case">Establish a principle rather than making decisions case by case</h2>
<p>An executive who makes many decisions is lazy and ineffectual – everything should be an example of a few rules.</p>
<p>When making decisions, focus on what is strategic and generic rather than trying to solve problems, or adapting to the apparent need of the current moment. Choose what is sound rather than what is clever.</p>
<p>The trickiest decision is between the right and wrong compromise.</p>
<h2 id="until-a-decision-has-been-acted-on-it-is-just-a-good-intention">Until a decision has been acted on it is just a good intention</h2>
<p>No task is complete until it has become part of organisational action or behaviour. When the pressures make the decision, this part is slighted.</p>
<p>In the military you give an order and it happens, but this is because all military organisations have learned long ago that the officer who has given an order goes out and sees for herself whether it has been carried out.</p>
<p>Failure to go out and look is the typical reason for persisting in a course of action after it has ceased to be appropriate.</p>
<h2 id="do-not-aim-for-consensus">Do not aim for consensus</h2>
<p>Effective executives create dissension and disagreement rather than consensus in order to make good decisions.</p>
<p>Disagreement means well thought out opinions on all sides. And the executive starts out with a commitment to find out why people disagree. First start with understanding. Only when you can understand the other party’s position can you be sure your position is right.</p>
<p>What do we have to know to test the validity of this hypothesis? What would the facts have to be to make this opinion tenable?</p>
<p>In meetings, one can either direct and listen or take part and contribute.</p>
<h2 id="effective-executives-think-about-what-they-can-contribute">Effective executives think about what they can contribute</h2>
<p>What can I contribute, that others cannot? Why am I on the payroll?</p>
<p>Results come from building on strength not shoring up weakness. Do what you are good at.</p>
<p>Ask how can I help, what can I do? How do my goals align with the goals of the organisation? Focus work on upward contribution. The executive who works at making strengths productive, works at making organisation performance compatible with personal achievement.</p>
<h2 id="making-your-boss-successful-is-the-best-way-to-progress">Making your boss successful is the best way to progress</h2>
<p>Making the strength of your boss productive is the key to your own effectiveness. It enables you to focus your contribution in such a way that it finds receptivity “upstairs”, and will be put to good use.</p>
<p>What does my boss do well, and what do they need from me to perform? Your boss is also human, with human weaknesses, that you can help shore up. And few things make an executive as effective as building on the strengths of their superior.</p>
<h2 id="you-do-not-need-exceptional-talent">You do not need exceptional talent</h2>
<p>This is not the leadership of brilliance and genius but the more modest leadership of dedication, determination and serious purpose.</p>
<p>Common people achieving uncommon results. It goes from mechanics, to attitudes, values and character; from procedure to commitment.</p>
<p>The task of the executive is not to change human beings but to multiply performance of everyone by putting strengths to work.</p>
<p>Put into the leadership position the person who can do the pace-setting job. As executives work towards being effective they raise the sights of their own and others and raise the performance level of the organisation.</p>
<h2 id="i-loved-this-book">I loved this book</h2>
<p>A lot of it really resonated with my own views about discipline, prioritisation and focusing on the strategic solution rather than solving problems on a case-by-case basis. It promises a utopian vision of not running around all the time fighting fires: “A well managed plant is a quiet place, a well managed organisation is a dull one.”</p>
Triggers2019-12-24T00:00:00+00:00https://www.annashipman.co.uk/jfdi/triggers.html<p><a href="https://www.marshallgoldsmith.com/product/triggers-creating-behavior-that-lasts-becoming-the-person-you-want-to-be/">Triggers</a> is a very interesting book. The subtitle says it’s about creating lasting behavioural change, which I did not find in my case, but it is full of useful, actionable insights about day-to-day issues we face at work and at home. Here are my notes.</p>
<h2 id="there-will-always-be-a-crisis">There will always be a crisis</h2>
<p>Marshall Goldsmith is an executive coach. He tends to work with a client on a particular issue, and at the beginning he says how long it might take (maybe a year, for example), but “I warn each client the process will take longer than they expect because there will be a crisis”.</p>
<p>I loved this; we always plan optimistically, but there is <em>always</em> a crisis.</p>
<h2 id="situational-leadership">Situational leadership</h2>
<p>He talks about the concept of situational leadership. When delegating tasks, there is a spectrum of four distinct styles.</p>
<ol>
<li><em>Directing</em> is for employees who need a lot of specific guidance to complete the task.</li>
<li><em>Coaching</em> is for employees who want and need to learn.</li>
<li><em>Supporting</em> is for employees who have the skill to complete the task but lack confidence to do it on their own.</li>
<li><em>Delegating</em> is for employees who score high on motivation, ability and confidence.</li>
</ol>
<p>This reflects a useful discussion I had with a previous mentor about how to delegate to different members of a team. I’ve <a href="/jfdi/delegating-to-a-team.html">previously written about delegating to a team</a> but I got hung up on how to manage how much I helped out once tasks had been picked up. My mentor at the time pointed out that treating everyone as an equal member of the team did not mean treating them all exactly the same; how much support is required varies from person to person and from task to task.</p>
<h2 id="it-is-very-insightful-about-the-balance-between-now-and-future">It is very insightful about the balance between now and future</h2>
<p>“We want short term gratification while we need long term benefit. And we never get a break from choosing one or the other. It’s the defining conflict of adult behavioural change.”</p>
<p>He quotes Drucker: “We are sacrificing the future on the altar of today.”</p>
<p>We choose to flirt with temptation rather than walk away – constantly testing ourselves against it – and dealing with the shock and distress when we fail. We have the impulse to always engage rather than selectively avoid.</p>
<p>(This reminded me of something I heard many years ago: “Do what you want to do, not what you feel like doing.”)</p>
<h2 id="aiwatt">AIWATT</h2>
<p>Am I Willing At This Time…</p>
<p>This is what Goldsmith suggests as a rule of thumb to “shrink your daily volume of stress, conflict, unpleasant debate, and wasted time. It is phrased in the form of a question you should be asking yourself whenever you must choose to either engage or ‘let it go’”.</p>
<p>Am I Willing At This Time… to make the make the investment required to make a positive difference on this topic?</p>
<p>He suggests this as a delaying mechanism to use to create a moment for rational thought between something that triggers something for us, and the resulting behaviour we are trying to change.</p>
<h2 id="aiwatt-for-not-sacrificing-the-future">AIWATT for not sacrificing the future</h2>
<p>I actually found it useful in a larger scale way.</p>
<p>At work, we are constantly presented with things that could be improved. We prioritise the things we are going to try to solve, and if we try and jump in on all of the issues we won’t make progress on the ones we’ve determined are the most important.</p>
<p>Anything we try and address does need some investment of time or effort; so I found this useful for not getting involved in things that I could see were problems and needed change but were not priorities.</p>
<h2 id="its-not-personal-its-business">It’s not personal, it’s business</h2>
<p>He talks about the parable of <a href="https://digital-dharma.net/buddhism/empty-rowboat/">the empty boat</a> to help you think about AIWATT.</p>
<p>Altogether less Zen is how I sometimes think of frustrating things that happen at work: <em>The Godfather</em>. It’s not personal, it’s business. (When I was looking up the exact quote for this post, I found that the scene I was actually thinking of was “Tell Mike it was only business. I always liked him.”).</p>
<h2 id="be-positive">Be positive</h2>
<p>He describes optimism as a “magic move”. People are automatically drawn to the confident individual who believes everything will work out.</p>
<p>Optimism almost makes the change process a self fulfilling prophecy.</p>
<p>He fines his clients for cynicism and sarcasm. He says we can be helpful, harmful or neutral, and complaining is not helpful.</p>
<p>He also recommends having some kind of positive trigger that you can refer to. He has a heartbreaking story about a photo that he keeps around to remind him that things aren’t that bad, to be grateful for what he has. He describes it as something that can act as a positive trigger in an otherwise negative environment.</p>
<h2 id="questions-for-one-to-ones-with-reports">Questions for one-to-ones with reports</h2>
<p>He had a suggestion of a really interesting structure for one-to-ones.</p>
<ul>
<li>Where are we going?</li>
<li>Where are you going?</li>
<li>What is going well?</li>
<li>Where can we improve?</li>
<li>How can I help you?</li>
<li>How can you help me?</li>
</ul>
<p>This was specifically for someone who had a chaotic leadership style and needed this kind of structure, but I thought these were really interesting questions to focus on. It makes you talk about some important areas you could easily gloss over otherwise.</p>
<p>I’ve written before about <a href="https://www.annashipman.co.uk/jfdi/line-management-questions.html">structuring one-to-ones</a>. In my current one-to-ones with reports we start by exchanging feedback: they give me one positive and one constructive piece of feedback, then I do the same for them. I like the idea of incorporating some of these other themes.</p>
<h2 id="thinking-about-what-to-stop">Thinking about what to stop</h2>
<p>Asking “what do we need to eliminate” fosters agreement more quickly than “what’s wrong”. The former triggers imagining a positive course of action. The latter triggers whining and complaining.</p>
<p>He talks about a “wheel of change”. You choose one thing to create, one to preserve, one to eliminate and one to accept. This helps focus the mind on the bigger changes you want to make.</p>
<h2 id="this-book-is-not-just-about-work">This book is not just about work</h2>
<p>As I mentioned in my notes on <a href="/jfdi/what-got-you-here.html">What Got You Here Won’t Get You There</a>, Triggers talks a lot about your life outside of work, and even talks about how to be happy; both of which I’ve not seen in other management books.</p>
<p>For example, he talks about how you might be professional at work, but “an amateur at home”; describing several stories about executives who were brilliant at work but poor spouses, parents or friends.</p>
<h2 id="active-questions-to-increase-happiness">Active questions to increase happiness</h2>
<p>He talks a lot about how being engaged in something, and being an active participant in trying to improve things, makes you happier and more productive. So for example, when asking employees whether they have set clear goals, active questions lead to better results. Instead of asking “Do you have clear goals?”, which invites people to pass the blame to the employer, ask “Have you done your best to set clear goals for yourself?”</p>
<p>He also suggests self-questioning, to help yourself stay engaged. For example, he suggests that in boring meetings, you regularly ask yourself the following questions:</p>
<ul>
<li>Did I do my best to be happy?</li>
<li>Did I do my best to find meaning?</li>
<li>Did I do my best to build positive relationships?</li>
<li>Did I do my best to be fully engaged?</li>
</ul>
<h2 id="daily-active-questions">Daily active questions</h2>
<p>His favoured approach for behavioural change is daily active questions. You decide what you want to change and then ask yourself at the end of every day, “did I do my best to…?”</p>
<p>He actually pays someone to phone him every night at 10pm to get his scores.</p>
<p>One example he gives is of someone who finds the sound of clinking ice cubes extremely annoying, such that he doesn’t enjoy a drink with his wife to relax. This person trains themselves to put up with it using daily questions. Other examples involve losing weight, or improving some aspect of work.</p>
<p>Some people favour self discipline (doing good things) over self control (saying no to bad things) and some the other way round – everyone has a preference. So phrasing daily questions a certain way can help.</p>
<h2 id="so-did-i-change">So, did I change?</h2>
<p>I did try the daily questions.</p>
<p>First I started out as he recommends, with about 10 questions. To start with, I did notice myself trying throughout the day to make the changes I had committed to, in order to be able to respond positively to that question in the evening. However, 10 questions on a 10 point scale was too many and felt like a drag, and after a week or so, the initial effect of wanting to give a good answer wore off.</p>
<p>I tried reducing the number of questions, and then changing it from a 10 point scale to a simple yes/no, but after some weeks, gave up on the project.</p>
<p>In summary, I don’t think his method worked for me.</p>
<p>There is something in it though; identifying what it is you want to change, working out how to remind yourself daily what those things are, and rewarding yourself when you do change.</p>
<h2 id="behavioural-change-aside-it-was-good">Behavioural change aside, it was good</h2>
<p>What I found interesting about this book was that it was basically a self-help book (which I don’t object to, but don’t generally reach for) slightly disguised as a strictly business book (which I am very into reading).</p>
<p>I didn’t agree with everything he said, but there were a lot of really interesting ideas about how to improve various aspects of one’s life. It was definitely worth my time.</p>
What got you here won't get you there2019-12-22T00:00:00+00:00https://www.annashipman.co.uk/jfdi/what-got-you-here.html<p>Earlier this year, I read <a href="https://www.goodreads.com/book/show/84525.What_Got_You_Here_Won_t_Get_You_There">What Got You Here Won’t Get You There</a>. Great title. For me, the book didn’t quite live up to my initial hopes, but it was interesting and thought-provoking. Here are my notes.</p>
<h2 id="the-book-is-about-bad-habits-that-stop-you-progressing">The book is about bad habits that stop you progressing</h2>
<p>Marshall Goldsmith is an executive coach, and this book is about the 20 behavioural habits that he sees preventing executives getting to the next level.</p>
<p>He quotes Peter Drucker: “Half the leaders I have met don’t need to learn what to do. They need to learn what to stop.”</p>
<h2 id="stop-adding-too-much-value">Stop adding too much value</h2>
<p>As a leader, you often know a lot about a piece of work, maybe more than the person you’ve delegated it to, and when they tell you your ideas for how to do it, you may try and add some value by suggesting an improvement to their plan.</p>
<p>However this is counter-productive because it is demotivating. “You may have improved the content of my idea by 5 per cent, but you’ve reduced my commitment to executing it by 50 per cent, because you’ve taken away my ownership of the idea… and I walk out of your office less enthused than when I went in.”</p>
<p>He tells a story about someone he coached who developed the habit of taking a breath before replying to anything, and once they’d got into that habit, they realised half of what they had been about to say was not actually worth saying.</p>
<h2 id="make-sure-you-are-clear-about-your-goals">Make sure you are clear about your goals</h2>
<p>When are you in a hurry? All the time. So it is all the more important to make sure you know why you are doing what you are doing.</p>
<p>You will always feel too busy or overcommitted.</p>
<p>(This reminded me of something I heard ages ago which I’ve found useful for things like agreeing – or not! – to do conference talks: don’t agree to do anything you wouldn’t do tomorrow. Because one day it will be tomorrow.)</p>
<p>He says part of being an effective leader is setting up systems to measure everything. That’s how you know how you are doing.</p>
<h2 id="retention-is-about-relationships">Retention is about relationships</h2>
<p>He talks about retaining good people and tells a story about changing someone’s role: “we would have done anything to keep her happy, we wanted to keep her”.</p>
<p>Your relationship with top talent is a strategic alliance rather than an employment contract.</p>
<p>Relatedly, he says that the Hawthorne effect is not just that people work better when the boss is watching, but also “it’s the reason entire factory floors work harder with greater morale when they see their bosses care about their welfare”.</p>
<h2 id="ask-for-feedback-often">Ask for feedback often</h2>
<p>Leaders who ask for input on a regular basis are seen as getting better. Remind people what you are trying to do by bringing up your objectives and asking “how am I doing?”</p>
<p>He also talks about what he calls “feedforward”; asking “what can I do to improve in the future?” This is a way colleagues can help each other.</p>
<h2 id="you-are-still-you-outside-of-work">You are still you outside of work</h2>
<p>Your flaws at work don’t vanish as soon as you get home.</p>
<p>He suggests you request feedback at home – what can I do to be a better partner/parent/friend?</p>
<p>This is a theme he goes more into in his subsequent book, <a href="https://www.marshallgoldsmith.com/product/triggers-creating-behavior-that-lasts-becoming-the-person-you-want-to-be/">Triggers</a>. I have not seen this in other management books but I like this approach, that both your home/personal life and work life are areas equally worthy of your investment to improve.</p>
<h2 id="how-to-recalibrate-if-you-find-your-staff-too-needy">How to recalibrate if you find your staff “too needy”</h2>
<p>He tells an interesting story about an executive who found they were spending too much time attending to requests for support from their staff. People are the most important part of the job, so if people feel they need your help, as a leader you want to be able to give it. However, as Camille Fournier points out in this excellent post, <a href="https://medium.com/@skamille/delegation-and-time-management-6cb326a880d3?">being helpful can actually be harmful</a>, and this executive needed to recalibrate.</p>
<p>In the story Goldsmith tells, the executive addressed it with two questions to their staff:</p>
<ul>
<li>“Your job: are there areas of your job where you think I am too involved or should be less involved?”</li>
<li>“My job: do you see me doing things that a person at my level should not be worrying about?”</li>
</ul>
<h2 id="some-suggestions-about-delegation">Some suggestions about delegation</h2>
<p>He suggests saying “I want you to do as much of my job as you can handle”.</p>
<p>You also have to remember you’re not managing you.</p>
<p>He quotes Drucker again: “know how to ask rather than tell”.</p>
<h2 id="follow-up-is-really-important">Follow up is really important</h2>
<p>He talks about an executive who asked for something to be done by sending a memo: Goldsmith asked them: how many read it, understood it, acted on it? The executive couldn’t answer.</p>
<p>For things that you want to get done, you need to do follow-up. Send the message. Ask people the next day if they heard it. Then ask if they understood it. Then a few days later ask if they did something about it.</p>
<h2 id="leadership-inventory">Leadership Inventory</h2>
<p>The book finishes with an appendix called the “Global Leadership Inventory”, part of a research project sponsored by Accenture into what characteristics high potential leaders have. There are 72 (grouped into areas) and I made a note of a few that resonated with me:</p>
<ul>
<li>Effectively involves people in decision making</li>
<li>Thrives in ambiguous situations (demonstrates flexibility where needed)</li>
<li>Sees change as an opportunity not a problem</li>
<li>Encourages creativity and innovation in others</li>
<li>Effectively translates creative ideas into business results</li>
<li>Demonstrates self confidence as a leader</li>
<li>Anticipates future opportunities</li>
<li>Understands competitive options available to customers</li>
<li>Effectively manages technology to increase productivity</li>
<li>Strives to arrive at an outcome with others</li>
<li>Builds an organisation where people care about the greater good</li>
<li>Achieves results that lead to long term shareholder value – holds people accountable for their results</li>
</ul>
<p>The whole appendix is an interesting list to return to when looking for ways to develop and grow.</p>
<h2 id="some-miscellanous-things-i-liked">Some miscellanous things I liked</h2>
<p>These don’t fit into a theme particularly, but some thoughts and ideas that I wanted to remember.</p>
<ul>
<li>People will do something only if it can be demonstrated that it is in their own best interests according to their own values.</li>
<li>Make a ‘to stop’ list instead of a ‘to do’ list.</li>
<li>Successful people stack the deck in their favour: the best employees, the best PAs, etc.</li>
<li>Forgiveness means letting go of the hope for a better past.</li>
</ul>
<h2 id="its-worth-reading">It’s worth reading</h2>
<p>Although I found this book didn’t fulfil my early expectations, it was definitely an interesting read and made me reflect on some aspects of my own leadership style.</p>
No next Next2019-09-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/no-next-next.html<p>Last week, I wrote about <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">how we worked out our tech strategy for FT.com</a>. In this post I talk about the strategy itself: No next Next. And also about the Sugababes.</p>
<h2 id="the-launch-of-next-hugely-improved-the-user-experience">The launch of “Next” hugely improved the user experience</h2>
<p>In October 2016, we replaced the old FT.com site with a new site, which we called “Next”.</p>
<p>The previous site didn’t have one team in charge of the whole; parts were owned by different parts of the business, meaning it was very hard to create a consistent user experience, keep it secure, or iterate quickly. Next is owned by one team which allows us to take a coherent approach to continuously improving the tech and customer experience.</p>
<p>The new site is built on a microservices architecture, ships hundreds of times a week and A/B tests everything. It is much faster than the old site and we demonstrated that <a href="https://medium.com/ft-product-technology/a-faster-ft-com-10e7c077dc1c">this directly impacted revenue</a>. It is much <a href="https://www.niemanlab.org/2016/10/the-financial-times-flips-the-switch-on-its-new-website-after-months-of-beta-testing-in-the-open/">better designed</a>, and our subscribers now come back more frequently and read more articles when they do. Earlier this year, we hit one million subscribers, a year ahead of our goal to do so.</p>
<h2 id="but-we-dont-want-to-have-to-do-it-again">But we don’t want to have to do it again</h2>
<p>Building Next to replace <a href="https://web.archive.org/web/20150226162405/http://www.ft.com/home/uk">the previous FT.com site</a> took took two years and cost in the order of magnitude of £10m. Building it also meant that we had very limited capacity to change the old site while that work was in progress, so as well as financial costs it also had opportunity costs.</p>
<p>So we need to make sure we make the most of that investment. In order to do that, we need to steer it so that we don’t make the same mistakes we made with the old site.</p>
<h2 id="no-next-next">No next Next</h2>
<p>In a phrase, our vision for FT.com is <em>No next ‘Next’</em>.</p>
<p>We must make sure that the product and the underlying technology do not drift so far off course that we have to rebuild FT.com again in a few years, spending entirely avoidable time and money.</p>
<p>Instead, we want our tech to be sustainable, supportable, and operationally reliable, and we want it to be simple enough for new starters to get to grips with.</p>
<p>We don’t want to spend time grappling with complex technology, we want to spend our efforts on adding new features, innovating, and continuously improving the subscriber experience and value that the FT brings.</p>
<p><img src="/img/side_by_side_ft.png" alt="Side by side of old site from 2016 and current FT.com front page" /></p>
<h2 id="the-ship-of-theseus-and-the-sugababes">The Ship of Theseus and the Sugababes</h2>
<p>If we want to make FT.com, the mobile apps, and other FT digital products sustainable for the long term rather than having to rebuild from scratch every few years, we will need to replace parts of it while it is in flight.</p>
<p>While talking about this we came up with a few analogies. <a href="https://en.wikipedia.org/wiki/Ship_of_Theseus">The Ship of Theseus</a>, rebuilt piece by piece while at sea. <a href="http://foolsandhorses.weebly.com/triggers-broom.html">Trigger sees his broom</a> as the same broom, despite the handle and head each being replaced a number of times. Or if those analogies don’t work for you, we could use the one that clicked it into place for <a href="https://aboutus.ft.com/en-gb/announcements/financial-times-appoints-john-kundert-as-chief-technical-officer-1/">our CTO</a>, the <a href="https://en.wikipedia.org/wiki/Sugababes#Members">Sugababes</a>, where the final line-up contained none of the original members.</p>
<p>Whether or not you think the final line-up of the Sugababes is really the Sugababes, that’s the approach we need to take with FT.com and our customer products to make them sustainable over the long term.</p>
<h2 id="replacing-the-first-band-member-is-already-increasing-performance">Replacing the first band member is already increasing performance</h2>
<p>In my <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">previous post on building a tech strategy</a>, I mentioned one way of finding out where to focus is asking engineers what code they feel afraid to touch. Through this method, we found that the package that manages the assets for FT.com hadn’t been worked on since the beginning of the Next project.</p>
<p>It performs multiple frontend services for FT.com, including building and loading client-side code, configuring and loading templates, as well as tracking, ads configuration and a lot more. It was tightly coupled to technical decisions made four years ago, and complicated. The current team did not feel confident making changes.</p>
<p>We put together a small team to update it, and they have now built a new service that splits all of this functionality into a set of loosely coupled, individually documented, and tested packages. The changes the team has made mean that any of our engineers can now confidently make contributions to this critical part of how we deliver the website.</p>
<p>It has also had an impact on the performance of the site. The team are currently migrating all of the FT.com services to the new packages, and each services that has been migrated shows decreases in the <a href="https://developers.google.com/web/tools/lighthouse/audits/time-to-interactive">Time To Interactive</a> of between 30% and 70%. This benefits users and also has an <a href="https://medium.com/ft-product-technology/a-faster-ft-com-10e7c077dc1c">impact on revenue</a>.</p>
<h2 id="communicating-to-the-team-and-the-rest-of-the-business">Communicating to the team and the rest of the business</h2>
<p>One of the purposes of a strategy is to give focus, so we identified the highest priority areas to get us back on track. The first thing we worked on was communication.</p>
<p>It’s all about communication. You can have the best vision and strategy in the world but if no-one knows about it it is not going to get you very far.</p>
<p>We have three main audiences for the tech strategy:</p>
<ol>
<li>Engineers working in our team, who need to know what the strategy is and what the priorities are so they can make sure they are working on the right things and can get involved with wider strategic work.</li>
<li>People outside of the team, so that they know what our focus is, how it will impact on their own priorities, and how to influence it.</li>
<li>Senior people in the business, for example the board, who need to know that by doing a little, often, we will not need to suddenly request another huge multi-year investment to rebuild the site again.</li>
</ol>
<p><img src="/img/no_next_next.png" alt="NO NEXT NEXT on a pink background" /></p>
<h2 id="communicating-the-tech-strategy-and-priorities">Communicating the tech strategy and priorities</h2>
<p>When we started working on the tech strategy we realised that there were communication channels missing so we had to fix that first. We also did some research into how people would like to receive the information.</p>
<p>We now do the following:</p>
<ul>
<li>In our monthly full team meeting, for all disciplines, we give a 3 minute update on our progress on the tech strategy.</li>
<li>I give a quarterly tech talk, open to anyone, about our progress against the tech strategy: what we’ve done and what we’re doing next.</li>
<li>We have a page about the tech strategy on our engineering wiki which we keep updated. Our wiki is in a private GitHub repo which means only people who have a GitHub license can see it, so we now run a nightly job that publishes it to a Heroku app that anyone at the FT can access.</li>
<li>Our team has a monthly newsletter, sent to anyone in the business who is interested, and we update that with the progress on the tech strategy.</li>
<li>When we have ad hoc updates about the tech strategy we email the team mailing list, and we always put [Tech Strategy] in the subject line so people can add filters (to make sure they never miss an update, of course!)</li>
<li>Finally, we are blogging about what we are working on, so you can expect to read more about it here!</li>
</ul>
<h2 id="refreshing-and-communicating-our-technical-principles">Refreshing and communicating our technical principles</h2>
<p>Tech principles are a high impact area to focus on because if they are well known and adhered to, that’s a way to affect everyone’s behaviour and help people move in the same direction.</p>
<p>When I joined the FT I found that there were technical principles, but there were four or five slightly different versions in different places, many of them in slide decks from presentations, and none of them actively used.</p>
<p>We’ve now refreshed our tech principles and publicised them. We’ve even made posters! <a href="https://medium.com/ft-product-technology/the-new-tech-principles-for-ft-com-dd788a896309">Matt Hinchliffe has written a great blog post with more detail about that here</a>.</p>
<p><img src="/img/tech_principles_poster.png" alt="A poster saying "Treat unblocking others as your priority"" /></p>
<h2 id="making-our-out-of-hours-process-sustainable">Making our out of hours process sustainable</h2>
<p>The number of production outages for FT.com and the apps is very low, but occasionally we do get called out of hours to handle incidents. The team on the out of hours support rota are very responsive and deal with issues quickly and well. However, because our tech has grown more complex over time, it’s hard to get people — especially new starters — to join the out of hours rota.</p>
<p>Part of the solution to this involves doing the hard work to make the tech simpler. But there are other ways to help with this. One important thing is to make sure things you might be called up to deal with at 3am are well documented, and to get our most crucial systems up to a good standard we ran a Documentation Day, which <a href="https://medium.com/ft-product-technology/documentation-day-how-the-ft-com-team-improved-our-documentation-to-95-usefulness-in-7-hours-b73d1a7e6f30">Jen Johnson has written a great blog post about here</a>.</p>
<p>We’ve also started running workshops to train people in dealing with out of hours incidents, which have really improved people’s confidence in being on the support rota. We will talk more about our progress on improving out of hours as we go.</p>
<h2 id="simplifying-our-tech-to-help-us-focus-on-growth">Simplifying our tech to help us focus on growth</h2>
<p>When systems get too complex, they become harder to support. It takes more time to identify and fix bugs, it is harder to spot potential security flaws, and engineers spend more time digging through code to try and understand it than they do on adding new features and improving the customer experience.</p>
<p>As the second law of thermodynamics states, everything tends towards entropy, so we have to actively work to fight this and keep our systems simple and high quality.</p>
<p>The effort we put into this pays off in reducing bugs and outages, increasing security, and making our technology easier to understand, meaning engineers can more easily work across the stack and spend their time on delivering things that add value to customers and to the business.</p>
<h2 id="measuring-the-simplicity-of-our-systems">Measuring the simplicity of our systems</h2>
<p>In order to reduce system complexity, it is very helpful to be able to measure it. Measurement will show us the most complex areas to tackle first, and when we start making improvements, we will be able to track our progress against the original complexity. Having a complexity measure is also important in communicating to people outside of engineering which areas need attention.</p>
<p>At the moment we are looking at visualising all our services and packages to indicate how well-maintained they are (for example, how frequently they are updated, how many open pull requests they have, etc). Later we will look at code complexity and other measures.</p>
<p>Our goal is to have an overview of our systems’ health that is meaningful to people outside engineering.</p>
<h2 id="clarifying-our-architecture">Clarifying our architecture</h2>
<p>We have multiple teams working on microservices who are empowered to make technical decisions.</p>
<p>However, one of the common problems of the <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">difficult teenage years</a> is that without a clear technical vision, the teams can get out of alignment with each other. Different teams can make small technical decisions that ultimately lead to conflicting or unclear architectural changes.</p>
<p>We are doing a piece of work firstly to clarify our current architecture and clearly identify pain points and conflicts, and then to work on decision-making. We want to make it possible for tech leads to make architectural decisions that are consistent with other teams’ technical decisions, without imposing too much process on the decision-making.</p>
<h2 id="joining-up-ftcom-and-the-mobile-apps">Joining up FT.com and the mobile apps</h2>
<p>When FT.com was rewritten, the team intentionally left the apps out of the rewrite so as not to take on too much at once, so “Next FT” is only the website, not the mobile apps. However, the team have been working hard since then on improving the apps and making sure features that are on the website are also on the apps, and vice versa.</p>
<p>The apps are a crucial part of our product, because in a modern news organisation we need to offer users many ways to consume our content, and it’s important to offer different things for different user needs. In addition, people who use the FT app tend to be very engaged with the FT, so there is a lot of value in improving their experience.</p>
<p>Because the apps were not in the Next rewrite, the app technology is out of step with the FT.com tech, and some of the underlying technology is now quite dated. We have been doing foundational work to bring the website development and app development closer together, so we can make use of the FT.com stack.</p>
<p>Once we’ve finished that we will be able to move much faster on the apps and do more exciting and ambitious work to add features for our highly engaged users.</p>
<h2 id="more-to-come">More to come!</h2>
<p>We will continue sharing our journey towards making our product more sustainable, and we’d love to hear your thoughts and experiences.</p>
<p>And if you are interested in joining us on this journey, <a href="https://roles.ft.com/">we are hiring</a>!</p>
<p><em>This post originally appeared on the <a href="https://medium.com/ft-product-technology/no-next-next-42c71541ebcc">FT Product & Technology blog</a></em>.</p>
The difficult teenage years: Setting tech strategy after a launch2019-09-02T00:00:00+00:00https://www.annashipman.co.uk/jfdi/difficult-teenage-years.html<p>When you are launching a new product to replace an existing one, it’s easy to rally behind the mission and make the right technical decisions that will get you over the line. But after the launch, it’s very common for things to lose focus: the vision is no longer so clear, corners you cut to get the product live come back to bite, and the tech can start to feel like it’s drifting off track.</p>
<p>This is the difficult teenage years.</p>
<h2 id="setting-a-strategy-to-get-back-on-track">Setting a strategy to get back on track</h2>
<p>We experienced this with the <a href="https://www.niemanlab.org/2016/10/the-financial-times-flips-the-switch-on-its-new-website-after-months-of-beta-testing-in-the-open/">launch of the new FT.com site in 2016</a>. The new site is much faster, responsive on different devices and uses modern technology. It has a microservices architecture and ships hundreds of times a week. The launch of the site led to a measurable increase in how often our subscribers visited the site and the number of articles they read while there, and it also led to a big increase in subscribers.</p>
<p>When I joined the FT in 2018 I found an excellent, smart and motivated team, good tech and a great culture. However, some things were drifting off track. This post is about how we created our tech strategy for getting out of the difficult teenage years, and how you could too.</p>
<h2 id="strategy-is-diagnosis-vision-and-a-plan">Strategy is diagnosis, vision and a plan</h2>
<p>It’s worth talking about <a href="https://www.annashipman.co.uk/jfdi/good-strategy-bad-strategy.html">what strategy is</a>. A strategy consists of three parts:</p>
<ol>
<li>Diagnosis: What’s the current situation?</li>
<li>Vision: What is your desired end state? Where are you trying to get to? What does ‘good’ look like?</li>
<li>The plan to get there: What are the highest impact steps you can take to get to your desired end state?</li>
</ol>
<h2 id="the-vision-for-the-launch-of-next">The vision for the launch of “Next”</h2>
<p>The <a href="https://web.archive.org/web/20151231154747/http://www.ft.com/home/uk">previous FT.com website</a> was a monolith that could only be released once a month, outside of working hours. The site itself was slow and it was also not <a href="https://en.wikipedia.org/wiki/Responsive_web_design">responsive</a>, so on mobile devices it looked like a tiny version of the desktop site. There also wasn’t one team responsible for the overall look, feel, and technical coherence of the site: different parts of the business owned different parts of the website.</p>
<p>Initially, a small team worked on a prototype of a new FT.com website, which they called “Next”. Next has a microservices architecture and is built in Node.js. The Next team focused on building a fast, responsive site, with an emphasis on shipping and measurement. Users were given the opportunity to opt in to the Beta, and 5% did, allowing the team to develop the site in the open with real users.</p>
<p>In October 2016 it was rolled out to all users, and “Next” became the current FT.com site.</p>
<h2 id="diagnosis-the-difficult-teenage-years">Diagnosis: The difficult teenage years</h2>
<p>When I joined the FT in 2018, the site was still good — fast, still using modern technology, still shipping hundreds of times a week, with a focus on measurement and A/B testing everything. The team were focused, smart and really friendly and inclusive.</p>
<p>However, there were some cracks starting to appear. Some of the types of comments I heard were:</p>
<ul>
<li>Engineers weren’t sure of the value of the work</li>
<li>There were areas of code people didn’t want to touch</li>
<li>There was duplication of code</li>
<li>A couple of services required deploying microservices in a certain order to make a change (which contradicts the point of microservices, as independently deployable units)</li>
<li>Feature changes felt too bitty</li>
<li>Some of the people who set technical direction had moved to other projects</li>
<li>Someone said, and this was echoed by different people in similar ways, “It doesn’t feel like we are owning or guiding a system, we are just jamming bits in”.</li>
<li>Finally, something I heard from different teams and different disciplines, a red flag that indicates a lack of focus: “We need more developers”.</li>
</ul>
<p>These kinds of comments show the main themes of the difficult teenage years:</p>
<ul>
<li>Lack of clear vision</li>
<li>The tech can start to feel like it’s drifting</li>
<li>Things aren’t communicated as well as they used to be</li>
</ul>
<p>This is very common after a launch.</p>
<h2 id="the-vision-after-the-launch-needs-to-be-different">The vision after the launch needs to be different</h2>
<p>You need a vision so you know what you are working towards, and to help communicate with others what you are working towards. When you are launching a new site to replace an existing site the vision is easy to communicate, because the new site will be better than the old one in a number of ways. You are replacing something that everyone can see the flaws with, as with the old FT.com site.</p>
<p>It is also easier to have conversations about what is in scope and out of scope before launch because there is usually a date in mind or some kind of milestone, so conversations about whether we have time to get something done before the deadline are much more focused.</p>
<p>After the launch, you no longer have the old site for comparison. So you need to be clear about what ‘good’ looks like.</p>
<h2 id="vision-no-next-next">Vision: No next Next</h2>
<p>The vision for our technology is that we make our tech sustainable.</p>
<p>Good is a healthy codebase that is simple enough for new starters to understand and for us to maintain, that is supportable and operationally reliable, delivers a good user experience and allows us to maintain and increase our ability to add value to the customer and the business.</p>
<p>Bad would be the tech drifting so far off track that we end up in a position where we have no choice but to do another rebuild; another “Next” project again in a few years, with all the costs associated with that.</p>
<p>So our vision is <em>No next ‘Next’</em>.</p>
<h2 id="the-plan-to-get-there-the-highest-impact-steps">The plan to get there: The highest impact steps</h2>
<p>Once you have the diagnosis and the vision, you need to work out what to do to get from where you are to where you want to be. What are the activities with the highest leverage? What are the things we should do now that will make other things easier later? What are the things that if we don’t do them now, other things we need to do later will be impossible?</p>
<h2 id="how-we-worked-out-what-to-do">How we worked out what to do</h2>
<p>Sometimes, once you are clear on the vision, it’s obvious what the next steps are. Often, however, you need to do some work to get there. Here are some things you can do to help:</p>
<ul>
<li>Listen to the patterns in what people say. When I joined the team, I had a one-to-one conversation with each of the engineers, to ask them what was going well, what was going badly (or was about to go badly) and whether there was anything they thought I should know. Some clear patterns emerged from those incredibly useful conversations.</li>
<li>If you have a specific questions, surveys can help. For example, we wanted to find out what areas of code people were most afraid of, so we sent out a survey to ask that question, and why.</li>
<li>For general patterns, the <a href="https://labs.spotify.com/2014/09/16/squad-health-check-model/">Spotify healthcheck</a> is very useful. We send this out every three months, and can see clear patterns which help us prioritise.</li>
<li>Early on, I gathered the senior engineers together and we had a session where you lay out cards on the floor to identify priorities. (<a href="https://www.annashipman.co.uk/jfdi/russells-strategy-advice.html">I explain a bit more about how to run that session here</a>). This is a really useful exercise for surfacing things you haven’t thought of, understanding dependencies, and making sure that you are all on the same page.
<img src="/img/card_workshop.png" alt="People laying out cards to identify priorities" /></li>
<li>Look for artefacts, for example technical principles, dashboards, architecture diagrams and runbooks. These are high leverage things that help make sure everyone has a shared understanding. If they are missing, getting some of those in place is likely to be an early priority. If they are in place, how frequently are they updated and are you sure they are current?</li>
<li>Finally, a live site is different from a site in development. There are areas that won’t have been a priority before the launch that are for a site in production. For example, how does out of hours support work? How do you handle critical security vulnerabilities? And how do you act on — or even get — customer feedback?</li>
</ul>
<h2 id="its-all-about-communication">It’s all about communication</h2>
<p>The most important part of a strategy is making sure people know what you’re doing and where you are going. You can have the best vision and best strategy but if no-one knows about it, it’s worthless.</p>
<p>Everyone in the team needs to understand where we are heading so they can make decisions about what they work on so we all get there.</p>
<p>People outside the team need to know our strategy so they can see how it helps the whole FT with our goals as a company, and so they can influence it if they have information that we need.</p>
<h2 id="communication-takes-a-lot-of-effort">Communication takes a lot of effort</h2>
<p>Communicating the vision and strategy involves communicating your message in lots of different ways: emails, presentations, small group discussions, posters, etc.
You have to say the same things over and over again, until you are bored of the sound of your own voice saying them, and then even more; because every time you say it, it will be new to someone. Perhaps they missed the last meeting, or weren’t listening, or are new. But if you want people to know what the strategy is, you need to keep talking about it.</p>
<p>Communication is harder after the launch because the vision may not be as clear, and some of the people involved in the launch will have moved on to other projects. So it’s important to put even more effort into how you communicate your vision and strategy.</p>
<h2 id="it-isnt-too-late-to-get-back-on-track">It isn’t too late to get back on track</h2>
<p>If some of this sounds familiar to you it’s worth letting you know that it might feel like it’s too late for you to get out of the difficult teenage years, but it isn’t. FT.com had been live for 18 months when I joined and we are doing well in getting back into a good shape. I hope that some of these techniques will help you, and I’d love to hear your stories about how you got your tech strategy back on track.</p>
<h2 id="more-detail-about-our-tech-strategy-is-on-its-way">More detail about our tech strategy is on its way!</h2>
<p>In my next post, I’ll talk in more detail about our <em>No next Next</em> tech strategy and how we are doing.</p>
<p>This is a blog post version of a talk I did at <a href="https://2019.continuouslifecycle.london/sessions/launch-difficult-teenage-years/">Continuous Lifecycle Conference London</a>. You can <a href="https://youtu.be/EkfqgQXfEv8">watch the video here</a>, or <a href="https://www.slideshare.net/annashipman/after-the-launch-the-difficult-teenage-years">see the slides here</a>.</p>
<p>And if you are interested in joining us on this journey, <a href="https://roles.ft.com/">we are hiring</a>!</p>
<p><em>This post originally appeared on the <a href="https://medium.com/ft-product-technology/the-difficult-teenage-years-setting-tech-strategy-after-a-launch-7f42eb94a424">FT Product & Technology blog</a></em>.</p>
After the launch: the difficult teenage years2019-08-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/after-the-launch.html<p>After you launch a new product and no longer have the launch to work towards, it’s very easy for things to drift off track. This is the difficult teenage years.</p>
<p>I gave a talk about how you can get out of the difficult teenage years at Continuous Lifecycle London in May this year.</p>
<p><a href="https://2019.continuouslifecycle.london/sessions/launch-difficult-teenage-years/">Talk description here</a>, and video and slides below.</p>
<h2 id="video-30-minutes">Video (30 minutes)</h2>
<div class="embedded">
<iframe src="https://www.youtube-nocookie.com/embed/EkfqgQXfEv8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
<h2 id="slides">Slides</h2>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/21py4uxqNIxMxn" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe>
Russell Davies's strategy advice2019-07-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/russells-strategy-advice.html<p>Last year, I met the excellent <a href="http://www.russelldavies.com/index.html">Russell Davies</a> to ask his advice on how to create a strategy. Russell was the Director of Strategy at the Government Digital Service while I was there. He and the internal comms team did an amazing job of making sure we were all aware of <a href="https://mikebracken.com/blog/the-strategy-is-delivery-again/">what we were doing</a> and why.</p>
<p>When I needed to create the tech strategy for my team at the Financial Times, Russell gave me some really useful advice, and has kindly agreed to let me share that here.</p>
<ul>
<li>The most important role of a strategy is to tell us what we are not going to do.</li>
<li>The strategy needs to be clear, concise and catchy.</li>
<li>A structure could be something like: 5 things we should do. Three will be obvious, one may be a surprise.</li>
<li>You need all versions of the strategy. The 30-second version, the 30-minute version and the full write-up. And then, the communication: mugs, <a href="https://www.flickr.com/photos/benterrett/11209009683">stickers</a>, posters, etc. Russell said “It will feel like propaganda; but no-one will notice” and this is true – while I was at GDS it did not feel like propaganda to me, it just felt like a really clear communication of what we stood for and what we were doing.</li>
<li>If you don’t have the channels to communicate your strategy, you may need to set them up. For example, if there is no team meeting, set one up – but make the first one about something other than the strategy, and then introduce it later. Russell is definitely a fan of getting everyone together, and even for a tech strategy this should include other disciplines, e.g. product, UX.</li>
<li>Don’t get too hung up on what’s strategy, what’s principles, etc.</li>
<li>Listen to what people say, what they want to do. Russell gave the example of <a href="https://www.gov.uk/guidance/government-design-principles#make-things-open-it-makes-things-better">“Make things open: it makes things better”</a>. Listen to what people say they want to do, and sometime when people are talking about it, someone will come up with a catchy phrase in conversation. Use that.</li>
<li>A strategy is a tool to allow you to say “if we do that, then we can’t do this”. Keep talking about it. Later, when something comes up that is against the strategy, people will recognise that and say “oh yes, you’ve been talking about that for years.”</li>
<li>Russell never goes into a quiet room and works out a strategy on his own. When working on strategy, he talks to people and they create it together.</li>
<li>He had a great suggestion of following a roadmap session that we used to do at GDS. Everyone writes down on index cards what should we do this year, and then place them on the floor – close to you if soon, further away if later. Then you all walk the floor together, talk about the cards, dependencies and likelihood of getting all those things done, and as you go you move things around or throw things out. We did this on my team when initially working out our tech strategy, and it was really useful for gathering ideas we hadn’t thought of, and having a discussion that meant we were all on the same page.</li>
<li>(<a href="https://twitter.com/tomskitomski">Tom Loosemore</a> ran this type of roadmap session a few times at GDS, and I talked to Tom about the nuts and bolts of how to run it. He said have a smallish group – no more than about 12 – start by outlining the goals, takes about a day. It needs someone very strict to run it: “Really?! Are we <em>really</em> going to do that in the next 3 months?!”. Once you get into the details of actually working out the roadmap for individual products or services, Tom also recommended Jamie Arnold’s <a href="https://www.jamiearnold.com/blog/2014/07/22/seven-questions-to-build-a-roadmap">Seven questions to build a roadmap</a>.)</li>
<li>When reviewing this post, Russell added something else about communicating strategy: “I once heard a story about Muhammed Ali’s trainer Angelo Dundee. As you can imagine, it was quite hard to tell Ali what to do so Dundee would just wait until Ali did the right thing and then praise it fulsomely. Communicating strategy might be a bit like that. Wait until people say something good about the strategy and then use that. (Where ‘waiting’ means ‘very carefully listening for’)”</li>
</ul>
<p>As well as some very useful advice, Russell gave me the book Good Strategy, Bad Strategy, which I have <a href="https://www.annashipman.co.uk/jfdi/good-strategy-bad-strategy.html">written some more notes on</a>.</p>
<p>He also has some more information and examples about strategy <a href="http://www.russelldavies.com/strategy.html">on his website</a>.</p>
How do you delegate to a group of people?2019-06-21T00:00:00+00:00https://www.annashipman.co.uk/jfdi/delegating-to-a-team.html<p>I’m the Technical Director of Customer Products at the Financial Times. In Customer Products, we run <a href="https://www.ft.com">FT.com</a> and the FT apps (as well as some other products, like <a href="https://ftalphaville.ft.com/">Alphaville</a>), and we have about 55 engineers.</p>
<p>I’ve got a team of five excellent principal engineers reporting to me, and my goal is for them to be running Customer Products without me. Ultimately, I’d like each of them to be a good candidate to take over my job.</p>
<p>However, one thing we’ve struggled with as a team is how they can figure out what they should be working on, and particularly, how best for me to delegate to them as a team rather than individually. This is about some changes we made to help with that.</p>
<h2 id="what-kind-of-things-am-i-delegating">What kind of things am I delegating?</h2>
<p>The things I need to delegate are cross-cutting, rather than domain-specific, so any one of them could pick them up. Examples of the kind of things that come in are: coordinate an evaluation of our testing strategy, lead on recruitment for a senior engineer role, work out what team should take on a piece of work and help define it, give an answer to this technical question from another team, and many others.</p>
<p>Ideally I want to be able to delegate anything that I am asked to do, as this is the best way for them to be running the group, and to get insight and practice at doing my job.</p>
<h2 id="how-i-initially-delegated-work">How I initially delegated work</h2>
<p>To start with, I delegated in one of three ways:</p>
<ol>
<li>We have a weekly meeting. In that meeting I might explain the item and someone would volunteer to pick it up.</li>
<li>I would send all of them a message (Slack/email) and say “Can someone pick this up” and someone would.</li>
<li>Occasionally – usually when things are very busy – I picked one of them myself and asked them individually to do something.</li>
</ol>
<p>This didn’t feel right. It got things done, but it meant that I was still in control of the flow of tasks and often even who did them.</p>
<p>I felt that there must be a better way to do this so that the principals know the problems we have that I need help with (and also know about the problems I don’t know about), without me doling them out.</p>
<h2 id="i-asked-for-advice">I asked for advice</h2>
<p>I have read a lot about delegation and had coaching, but that has tended to focus on how to delegate one piece of work to one person. I couldn’t figure out how best to delegate to a whole team. So I reached out to my network and asked them for advice and to hear some of their experiences around what had worked and what hadn’t.</p>
<p>I particularly asked for people’s experience around the practical aspects – types of/frequency of meetings, how/when people used email groups, etc.</p>
<p>I got lots of extremely useful suggestions from people. Here are some of the suggestions we had, what we tried and how it’s going for our team.</p>
<h2 id="some-people-suggested-a-more-kanban-approach">Some people suggested a more Kanban approach</h2>
<p>Some people suggested using an approach as I would in a delivery team. For example, weekly/fortnightly kick-offs, Kanban board, stand-ups, etc.</p>
<p>I was initially resistant to this. After all, we are not a delivery team, and part of the job is juggling many things at once; most of the things we do can’t be worked on until they are done. They are often ‘prepare for something in the future then wait for that thing to happen then do something else’, or ‘ask some people to do some work and keep an eye on it’.</p>
<p>However, we decided to try having a physical wall to see if that helped. Rather than a normal Kanban wall, we have columns for each of our names, and rows for ‘on their radar’, ‘working on’ and ‘done’. Next to that we have three other areas for cards: ‘Upcoming considerations’, ‘To pick up’ and ‘Not looking at’.</p>
<p><img src="/img/stand-up.jpg" alt="Five people in a stand-up" /></p>
<p>(c) <a href="https://www.ft.com/luke">Luke Kavanagh</a></p>
<p>We have a weekly stand-up, on Fridays, which usually lasts half an hour. We initially tried it as an experiment, and quickly found that it was really helpful. I’ve also found the wall extremely useful, despite my initial reservations. It works really well to be able to see that something is being worked on and who is working on it.</p>
<p>We also added retrospectives, once every six weeks, in which we reflect on just our processes within the team of principal engineers and make improvements.</p>
<h2 id="i-wasnt-sharing-all-the-context">I wasn’t sharing all the context</h2>
<p>Other people usefully pointed out that because of my role, I have a huge amount of context not available to my team.</p>
<p>For example, <a href="https://niksilver.com">Nik Silver</a> gave me this insight: “By virtue of you being in the position you are you attend a wide variety of meetings, you meet and listen to all manner of stakeholders, you weigh up and synthesise what you hear and you make judgements accordingly. None of your team are in all the same meetings you are, listening to those stakeholders, and therefore they can’t reasonably create the same backlog of things to do.”</p>
<p>As well as being in those meetings, I was also being sent emails on a wide range of topics and copied in on lots of email threads where they wanted engineering input from Customer Products.</p>
<h2 id="why-i-didnt-share-context">Why I didn’t share context</h2>
<p>In fact, not sharing context was a deliberate decision on my part. I get <em>a lot</em> of emails and go to a lot of meetings. A lot of this is requests to do things that I don’t think are a priority at the moment, so it didn’t seem to make sense to me to pass all that on.</p>
<p>I also didn’t want to drown my team in emails, because it’s hard to get stuff done when you are getting lots and lots of emails.</p>
<p>However, the downside of not passing this on was that I was denying them the opportunity to get all the context I had and make their own decisions about what was a priority; and it also meant that I became a blocker for information to pass through. If someone went directly to them about the same issue, they wouldn’t even know that I’d stopped it earlier.</p>
<p>Essentially I thought I was protecting them and helping them do their jobs, but in fact I was actually blocking the information flow and making it harder for them to do their jobs.</p>
<h2 id="i-asked-them-if-they-wanted-the-context">I asked them if they wanted the context</h2>
<p>I explained the situation to the team and some of the recommendations, and they said that they would prefer to get lots of emails and have the context, rather than have me policing it.</p>
<p>So we made some immediate changes:</p>
<ul>
<li>We set up a mailing list for the principal engineers and started publicising that as who to email, rather than emailing me only.</li>
<li>I started sending a lot more emails to that email address. Sometimes this would be forwarding an email I’ve received, adding “FYI”, so they have the context. If I had a bit more info I might add it in the email.</li>
<li>I look for opportunities for principals to go to meetings as my delegate. After the meeting they share back notes to the rest of the team, including me.</li>
<li>If I feel I need to be in a meeting where useful context will be shared but it’s possible to add someone, I ask one of them to accompany me, with the purpose of taking notes and sharing context with the rest of the principal engineers and also getting in practice at those sorts of meetings, and meeting the stakeholders.</li>
<li>Where it hasn’t worked for someone to come to the meeting with me (perhaps a 1:1), I write up notes and email them to the group, or talk through them at our weekly meeting.</li>
<li>We already had a short weekly meeting, but we extended it to 50 minutes. We now mark items on the agenda as ‘Proposal’ (ideally will have been circulated in advance), ‘Discussion’, or ‘Info’.</li>
</ul>
<h2 id="the-team-solve-problems-before-ive-even-heard-about-them">The team solve problems before I’ve even heard about them</h2>
<p>From my perspective a lot of these changes are going really well. Quite often, someone will email or mention at stand-up a problem that they have picked up and solved before I’ve even heard that it exists.</p>
<p>It also seems to be helping balance work a bit better – things sometimes get passed to others at stand-up if someone has too much on their plate.</p>
<p>People email our group email address rather than me or individuals in the team much more often, which means we all have a lot more context, and we can respond to things much faster as questions don’t get lost in someone’s inbox.</p>
<h2 id="making-sure-we-are-working-on-the-right-stuff">Making sure we are working on the right stuff</h2>
<p>These changes meant that we made huge improvements in getting things done, but we started to wonder whether we were always thinking about whether the things we were doing more efficiently were the right things. We felt that we might be being too reactive, and not necessarily always working on the highest priority or most strategic pieces of work.</p>
<h2 id="colour-coding-types-of-work">Colour-coding types of work</h2>
<p>We have taken two approaches to this. Firstly, we decided to be more conscious of the type of things we are working on. We think there are three categories of work we should be doing:</p>
<ol>
<li><strong>Team health</strong> e.g. making sure line management is working, improving the dev huddle, helping to create opportunities for engineers on our team to grow their careers, helping with recruitment, etc.</li>
<li><strong>Technical oversight</strong>, e.g. using our expertise as senior technologists to support team members with technical issues, helping with decisions that might affect the architecture of a number of teams, overseeing pieces of work with other teams at the FT, etc.</li>
<li><strong>Technical strategy</strong>: mainly this is working on the defined next steps of our tech strategy, but this also includes strategic things like writing blog posts, or arranging workshops.</li>
</ol>
<p>We then colour-coded the cards we use on our wall. The idea is that we should be spending around a third of our time on each of those things.</p>
<p>Because we are still using cards on a wall, we can’t measure how much time we are spending, but we can at a glance make sure that the team in general, and each of us individually, are doing roughly a split of those those types of task.</p>
<p>It’s also turned out to be really useful for making sure that certain people weren’t always doing certain things, e.g. for making sure that the women on the team do not always end up doing the team health tasks.</p>
<p><img src="/img/colour-coded-cards.jpg" alt="Four cards on a board, two coloured blue, one yellow, one pink" /></p>
<h2 id="regular-planning-meetings">Regular planning meetings</h2>
<p>We’ve also recently added planning meetings. While we do have a weekly meeting, we don’t always make sure that we cover upcoming concerns. We haven’t quite worked out the best format for the planning meetings yet, but so far it’s involved discussing a list of things that are coming up that we are not sure are being addressed in other ways; for example things that we don’t think are covered by our tech strategy and possibly should be.</p>
<p>A lot of the work in the planning meeting is ahead of time in compiling the agenda. This has been useful so far, and we will tweak the format to make it more useful as we go on.</p>
<h2 id="sharing-my-thinking">Sharing my thinking</h2>
<p>I have a clear idea of where we are trying to get to with our tech strategy and what good looks like, but I don’t think I’ve shared that consistently well, so I’ve been consciously working on communicating that better.</p>
<p>For example, I’ve added a page to the internal wiki, I gave <a href="https://continuouslifecycle.london/sessions/launch-difficult-teenage-years/">a talk at a conference</a> and I’m currently working on a blog post (watch this space!). I’m also working on how we actually measure/evaluate that our tech strategy has been successful.</p>
<p>Interesting suggestions I had from my network around this included the idea of <a href="https://en.wikipedia.org/wiki/Intent_(military)">command intent</a> and Amazon’s practice of <a href="https://www.allthingsdistributed.com/2006/11/working_backwards.html">working backwards</a>.</p>
<p>As a team, we could also do more to share what we are working on with the rest of Customer Products. We’ve let people know that they are welcome to attend our stand-ups, but so far no-one has, and we could do more to let people know what we’re working on.</p>
<h2 id="these-changes-have-really-improved-how-we-work-together">These changes have really improved how we work together</h2>
<p>We are now working together and sharing information really well as a team, and others are approaching the team with questions, rather than approaching individuals.</p>
<p>We’ve got room for improvement, but it’s definitely going in the right direction. Recently, I’ve been out of the office for a couple of weeks. When I came back, everything had carried on. Decisions had been made, new work had come in and been handled, and apart from some budget approvals, there was nothing that had been on hold waiting for me to get back. This is a brilliant result.</p>
Good Strategy, Bad Strategy2019-05-17T00:00:00+00:00https://www.annashipman.co.uk/jfdi/good-strategy-bad-strategy.html<p>A while ago, the excellent <a href="http://www.russelldavies.com/index.html">Russell Davies</a> gave me a copy of <a href="http://goodbadstrategy.com/">Good Strategy, Bad Strategy</a>. He said it was the best book he’d read on the topic, and I found it extremely useful. Here are some of my notes, but I recommend reading it.</p>
<h2 id="strategy-is-diagnosis-vision-and-a-plan">Strategy is diagnosis, vision, and a plan</h2>
<p>The kernel of a strategy contains 3 things:</p>
<ol>
<li>Diagnosis: the current situation</li>
<li>Vision: where you want to get to; this is your guiding policy</li>
<li>Coherent action: how you get from your current situation to your desired end state. Coherent action consists of feasible plans, resource commitments and actions.</li>
</ol>
<p>The core of strategy is discovering the critical factors and designing a way to coordinate and focus on actions to deal with them, including risk mitigation.</p>
<p>Good strategy is not just what you are trying to do, it’s also why and how.</p>
<h2 id="good-strategy-almost-always-looks-simple">Good strategy almost always looks simple</h2>
<p>The book opens with a description of The Battle of Trafalgar, and how the British (led by Lord Nelson) won, even though there were fewer British ships than French and Spanish ships.</p>
<p>Instead of following what was the usual tactic at the time, approaching in a single line, Nelson’s fleet approached in two columns, one aiming at the centre of the French and Spanish line, in order to break up their formation. However, this did put the ships at the front of the columns in greater danger.</p>
<p>In summary, his strategy was to risk his lead ships to break the coherence of his opponents’ fleet.</p>
<p>Good strategy almost always looks this simple.</p>
<h2 id="what-strategy-is-not">What strategy is not</h2>
<p>The author talks a lot about what is generally described as strategy and explains why it’s not. For example:</p>
<ul>
<li>Ambition is drive and zeal to excel</li>
<li>Determination is commitment and grit</li>
<li>Innovation is the discovery and engineering of new ways to do things</li>
<li>Inspirational leadership motivates people to sacrifice for their own and the common good</li>
</ul>
<p>And strategy, responsive to innovation and ambition, selects the path; identifying how, where, and why determination and leadership are to be applied.</p>
<h2 id="a-strategy-is-like-a-lever-that-magnifies-force">A strategy is like a lever that magnifies force</h2>
<p>Leaders must identify the critical obstacles to forward progress and develop a coherent approach to overcoming them.</p>
<p>The plan should be the highest impact areas. What single feasible objectives will make the biggest difference?</p>
<p>A strategy coordinates action to address a specific challenge. The job of the leader is to create the conditions that will make that push effective; to have a strategy worthy of the effort called upon.</p>
<p>It’s not enough just to focus – we need to think about why that is the focus. We need to apply power to the right target.</p>
<h2 id="it-involves-making-hard-choices">It involves making hard choices</h2>
<p>Creating a strategy involves choice, and the difficult work of casting out other things.</p>
<p>It is problem solving. By its very nature, you need to make hard choices. You need to address the elephant in the room.</p>
<p>Good strategy usually emphasises focus over compromise.</p>
<p>Universal buy-in means a choice hasn’t been made.</p>
<h2 id="diagnosis">Diagnosis</h2>
<p>Diagnosis should replace the overwhelming complexity of reality with a simpler story that calls attention to its crucial aspects.</p>
<p>The diagnosis part of the strategy is handing the organisation a problem it can solve.</p>
<h2 id="working-on-the-strategy">Working on the strategy</h2>
<p>The disconnect between current results and current action is what makes strategy hard and interesting.</p>
<p>A good strategy is a hypothesis about what will work formed by educated judgement. Exploit your rivals’ weaknesses and avoid leading with your own.</p>
<p>A strategy should be episodic, though not necessarily annual.</p>
<h2 id="you-need-to-make-your-strategy-robust">You need to make your strategy robust</h2>
<p>You have to be able to defend your strategy.</p>
<p>He has some interesting suggestions about how you can question your own judgement. For example have in your head a panel of people that you know what kind of thing they’d say, and imagine them critiquing your strategy.</p>
<p>He also suggests that you note down judgements you make over time, and then refer back to improve your process.</p>
<h2 id="it-involves-constant-work">It involves constant work</h2>
<p>Making such a policy work takes more than a plan on paper – you need to work to maintain the coherence of the plan, every quarter, year, decade.</p>
<p><a href="https://twitter.com/iamdeepa/status/944237019265298432">This Twitter thread</a> from Deepa Subramaniam is also very useful, particularly the practical tips on how to do that work.</p>
<h2 id="you-should-read-good-strategy-bad-strategy">You should read Good Strategy, Bad Strategy</h2>
<p>These are some of the notes I’ve found useful to refer back to. <a href="https://www.slideshare.net/sophiedennis/usercentred-digital-strategy-ux-in-the-city-manchester-2017">These slides</a> by <a href="https://twitter.com/sophiedennis">Sophie Dennis</a> are also an excellent summary of some of the main points.</p>
<p>But the book itself is extremely worth reading. It has made a big difference to how I think about and work on strategy.</p>
Finance for non-accountants2019-04-23T00:00:00+00:00https://www.annashipman.co.uk/jfdi/finance-for-non-accountants.html<p>This is the first role I’ve had managing a large budget and I recently had some excellent training from my finance director colleague <a href="https://www.linkedin.com/in/isabelle-campbell18/">Isabelle Campbell</a>. Here are my notes.</p>
<h2 id="the-role-of-the-finance-team-in-an-organisation">The role of the finance team in an organisation</h2>
<p>Isabelle started by answering the question “what is even the point of a finance team?”</p>
<p>There is a lot to know about finance. The current regulations are on dictionary-thin paper and stack up about four feet high. So the finance team takes care of keeping on top of regulations, tax and the latest accounting standards, as well as reporting.</p>
<p>They do budgeting and forecasting, and can also advise on strategic decisions.</p>
<h2 id="accounting-years-are-important">Accounting years are important</h2>
<p>The first non-obvious thing I learned was that the concept of an accounting year is very important. This is because a profit & loss statement (P&L) is only for the current accounting year. So whether something is capital or operating expenditure (more on that in a moment) is defined by the value within or beyond the accounting year.</p>
<p>Most UK companies use the calendar year (i.e. January 1st to December 31st). Many instead follow the tax year (so April 6th to April 5th; or April 1st to March 31st). In the UK you can set your accounting year however you like.</p>
<h2 id="annual-accounts">Annual accounts</h2>
<p>Every accounting year, most UK companies have a requirement to produce annual accounts. This can include a profit and loss statement (a P&L), a balance sheet and a cash flow statement of varying levels of detail and complexity.</p>
<p>We spent more time learning about the P&L rather than the other two, so I’ll talk more about that, but in brief:</p>
<ul>
<li>the balance sheet is a statement of the company’s assets and liabilities</li>
<li>the cash flow statement is a statement that shows how changes in the balance sheets and P&L affect the cash. The cash flow statement is essentially a check on the balance sheet and P&L.</li>
</ul>
<h2 id="how-to-read-a-pl">How to read a P&L</h2>
<p>For example, here is John Lewis’s P&L for last year. It was an image in a PDF, so I’ve typed out the contents below.</p>
<p><img src="/img/john-lewis-pl.png" alt="Screenshot of John Lewis P&L, text is below this image" /></p>
<table class="underline-table">
<tr>
<th>
Notes
</th>
<td>
</td>
<th>
2018 <br />
£m
</th>
<th>
2017 <br />
£m
</th>
</tr>
<tr>
<td>
1.2 2.1
</td>
<td>
<b>Gross sales</b>
</td>
<td>
11,597.7
</td>
<td>
11,374.2
</td>
</tr>
<tr>
<td>
2.1
</td>
<td>
<b>Revenue</b>
</td>
<td>
10,204.0
</td>
<td>
10,026.2
</td>
</tr>
<tr>
<td>
</td>
<td>
Cost of sales
</td>
<td>
(6839.5)
</td>
<td>
(6633.1)
</td>
</tr>
<tr>
<td>
</td>
<td>
<b>Gross profits</b>
</td>
<td>
3,364.5
</td>
<td>
3,393.1
</td>
</tr>
<tr>
<td>
</td>
<td>
Other operating income
</td>
<td>
111.3
</td>
<td>
92.6
</td>
</tr>
<tr>
<td>
2.2
</td>
<td>
<b>Operating expenses before exceptional items and Partnership Bonus</b>
</td>
<td>
(3,114.0)
</td>
<td>
(3,007.8)
</td>
</tr>
<tr>
<td>
3.3
</td>
<td>
Share of (loss)/profit of joint venture (net of tax)
</td>
<td>
(1.0)
</td>
<td>
0.3
</td>
</tr>
<tr>
<td>
2.1
</td>
<td>
<b>Operating profit before exceptional items and Partnership Bonus</b>
</td>
<td>
360.8
</td>
<td>
478.2
</td>
</tr>
<tr>
<td>
2.3
</td>
<td>
Exceptional items
</td>
<td>
(111.3)
</td>
<td>
171.2
</td>
</tr>
<tr>
<td>
2.1
</td>
<td>
Operating profit before Partnership Bonus
</td>
<td>
249.5
</td>
<td>
649.4
</td>
</tr>
<tr>
<td>
5.1
</td>
<td>
Finance costs
</td>
<td>
(85.7)
</td>
<td>
(109.7)
</td>
</tr>
<tr>
<td>
5.1
</td>
<td>
Finance income
</td>
<td>
141.1
</td>
<td>
1.9
</td>
</tr>
<tr>
<td>
</td>
<td>
<b>Profit before Partnership Bonus and tax</b>
</td>
<td>
177.9
</td>
<td>
541.6
</td>
</tr>
<tr>
<td>
</td>
<td>
Partnership Bonus
</td>
<td>
(74.0)
</td>
<td>
(89.6)
</td>
</tr>
<tr>
<td>
2.4
</td>
<td>
<b>Profit before tax</b>
</td>
<td>
103.9
</td>
<td>
452.2
</td>
</tr>
<tr>
<td>
2.7
</td>
<td>
Taxation
</td>
<td>
(29.8)
</td>
<td>
(98.7)
</td>
</tr>
<tr>
<td>
</td>
<td>
<b>Profit for the year</b>
</td>
<td>
74.1
</td>
<td>
353.5
</td>
</tr>
<tr>
<td>
2.1
</td>
<td>
<b>Profit before Partnership Bonus, tax and exceptional items</b>
</td>
<td>
289.2
</td>
<td>
370.4
</td>
</tr>
</table>
<p>There will be a scale. At the top, we can see the scale is millions (“£m”).</p>
<p>If a sum is in brackets, it’s negative. For example, “Cost of sales” in 2018 was “(6839.5)”, that means it was -£6,839,500,000, i.e. it cost them ~$7bn to buy or produce what they sold. (This may not include all costs – to see what all the parts are, we’d have to refer to note 2.1.)</p>
<p>The bottom line (hence the phrase!) is the revenue minus expenditure, i.e. the profit.</p>
<p>In this example the bottom line is £289.2m, though taxes, bonuses and exceptional items will come out of that. Somtimes the bottom line may be used to just indicate gross profit, i.e. before any deductions at all.</p>
<h2 id="capex-and-opex">Capex and opex</h2>
<p>Capex is capital expenditure. This means money spent on something that will have a longer term value; specifically this means a value outside of this accounting year. Buying a car for £30,000 would probably be capex, because you expect that car to last longer than a year.</p>
<p>Capex is money you spend on buying assets.</p>
<p>Opex is operational expenditure. This is money you spend on things that do not retain value for the company. For example, if you lease that car for £8,000 per year, that’s opex. There’s no value to the company. When you stop paying, you no longer have the car. Examples of opex include paying salaries and rent, buying stationery, etc.</p>
<p>Opex is the cost of operating the business in the year.</p>
<h2 id="capex-and-opex-in-the-21st-century">Capex and opex in the 21st century</h2>
<p>All reasonably straightforward when talking about buying vs leasing cars. With some things it is obvious whether something is capex (for example, buying a fleet of cars) or opex (renting a garage).</p>
<p>But there are a lot of grey areas. For example, what if our car had actually cost £8,000 to buy? Should we capitalise that if we expect it to last more than a year, or should we in fact just consider it part of this year’s operating expenses? There are guidelines, but this kind of thing is a judgment call and something the finance department can help with.</p>
<p>It becomes even more complicated when you get into the kinds of work most companies do now.</p>
<p>For example, staff costs such as salary are generally opex. But if they are working on something capitalisable, like software, then there is a question there. Say you have a team of people working on building an A/B testing framework that you expect will be used for the next 5-10 years. That A/B testing framework is an asset that you could potentially capitalise, in which case you may be able to capitalise some or all of the staff costs.</p>
<h2 id="capex-is-not-included-in-the-pl">Capex is not included in the P&L</h2>
<p>Capex and opex are treated in different ways. One extremely influential distinction is that capex does not appear on the P&L. Instead, it appears on the balance sheet as an asset.</p>
<h2 id="technology-is-usually-a-cost-centre">Technology is usually a cost centre</h2>
<p>A cost centre is a part of the business to which you allocate costs and is not responsible for revenues. This is in contrast to a profit centre, which can add to the company’s costs and profits.</p>
<p>In most businesses, technology is a cost centre, meaning that the main way the technology department can increase the overall profitability is by cutting costs, and/or making the revenue-generating areas of the business more economical or efficient.</p>
<p>Cost centres generally do not produce their own balance sheet.</p>
<h2 id="this-can-lead-to-misalignment-of-incentives">This can lead to misalignment of incentives</h2>
<p>Focusing on a P&L rather than having full visibility of what sits on the company’s balance sheet can potentially lead to incentives that are not aligned.</p>
<p>A big example in software is buy vs build. In general, you should build things that represent your core competency and things that will give you a competitive edge, and you should buy everything else. So if you’re not a hosting provider, use cloud computing for your infrastructure and hosting.</p>
<p>However, that cost is not capitalisable. The company does not gain any asset from paying AWS or Heroku. This means the costs will appear on your P&L. If, instead of using cloud computing, you bought your own hardware and hired a team of people working full-time in a data centre you own, that would be very capitalisable and the costs would not appear on your P&L.</p>
<p>It’s clearly a bad idea, for a number of other reasons, but the incentives here can be poorly aligned.</p>
<h2 id="amortisation-depreciation-and-write-downs">Amortisation, depreciation and write-downs</h2>
<p>Capitalised assets appear on the balance sheet. However, normally they lose value over time. This is called depreciation, and the annual cost of this is deducted from the balance sheet and charged to the P&L.</p>
<p>In the UK, ‘depreciation’ refers to the loss of value on tangible assets. There is also ‘amortisation’, which in the UK usually refers to loss of value of intangible assets, like brand. (US companies usually call depreciation amortisation, which can be confusing).</p>
<p>Sometimes assets lose more value than has been allowed for in a pre-determined depreciation or amortisation rate. In this case, the company might “write down” the cost, which means further reducing the value on the balance sheet. The amount it is reduced by is charged to the P&L. Whether to do this is usually a judgment call, though a company may receive pressure from their auditors.</p>
<h2 id="impairment-and-goodwill">Impairment and goodwill</h2>
<p>A recent big example of a write-down was at the end of last year, when <a href="http://www.cityam.com/270434/us-telecoms-company-verizon-announces-46bn-writedown-oath">Verizon took an impairment charge of $4.6bn on their purchase of Oath</a> (the company formed from Yahoo & AOL).</p>
<p>When a company buys another company, they have an expectation of the value that the asset will generate, as with any asset. If the purchased company turns out not to be delivering the value, for example due to changes in the market, government regulations, loss of brand reputation etc, then the asset is no longer worth what it was, and the owning company’s balance sheet suffers.</p>
<p>Part of the purchase amount agreed is the value of the assets. The remainder of the cost of the company is what is called “goodwill”: this is the difference between the assets that you can put a value to and how much you are willing to pay for it.</p>
<p>In the case of Verizon and Oath, the $4.6bn was a “non-cash goodwill impairment charge”, meaning the “goodwill” aspect of the deal was no longer considered to be worth what Verizon had paid, so they were taking a “write-down”, i.e. reducing the overall value of Verizon’s assets by $4.6bn, a figure which would have come out of Verizon’s 2018 P&L.</p>
<h2 id="budgets-and-forecasts">Budgets and forecasts</h2>
<p>The P&L budget for the year is worked out towards the end of the previous year. It covers everything that might happen in the business in the next year. How many employees the company is likely to have, how much revenue from existing and new customers can be expected, etc. It also involves asking the question of what can be capitalised.</p>
<p>A forecast can be done at any point during that year, and takes into account new information. For example, you might do a Q1 (Quarter 1, i.e. the first three months of your accounting year) forecast at the end of March. Knowing what we know now about what’s happened in the first three months of the year, what do we know about the rest of the year? Forecasts might be a few months, or a few years.</p>
<p>A forecast may not always change the budget; so a Q1 forecast that shows a higher or lower than predicted revenue may not result in a revised budget, but it does mean at least that you are armed with the information.</p>
<h2 id="how-to-find-a-pl-to-read">How to find a P&L to read</h2>
<p>If you want to browse the accounts of companies, you can! <a href="https://www.gov.uk/get-information-about-a-company">https://www.gov.uk/get-information-about-a-company</a> will take you to the Companies House search (currently in Beta). I searched for John Lewis Partnership, and then ‘Filing history’. The image above is from page 85 of their company accounts. Quite a useful tip if you want information on a company you are thinking of joining, or even if you’d like to know more about the company you work at.</p>
<p>Smaller companies only have to file restricted information in their accounts, but there is still interesting information about them on Companies House, e.g. who the directors are.</p>
<h2 id="this-has-only-scratched-the-surface">This has only scratched the surface</h2>
<p>There were loads of other things that we didn’t cover, and given that the regulations are four feet high, may not ever cover, but it was a really useful session and has given me much a better grasp of some of the competing priorities we need to think about.</p>
How to make the most out of meetings with vendors2019-03-22T00:00:00+00:00https://www.annashipman.co.uk/jfdi/meeting-vendors.html<p>Building relationships with vendors is an important part of my role as Tech Director and when I joined the Financial Times, the excellent <a href="https://twitter.com/rtshilston">Rob Shilston</a> gave me his five top top tips on how to make the most of those meetings. He has kindly agreed to share them here.</p>
<h2 id="1-use-it-as-an-opportunity-to-learn-about-how-others-are-doing-things">1. Use it as an opportunity to learn about how others are doing things</h2>
<ul>
<li>“What changes in investments are you seeing other organisations making?”</li>
<li>“How have others turned this tech into a business advantage?”</li>
<li>“What’s really impressed you recently?” and “We like to learn from other teams who are doing great things. Who would you recommend we speak with?”</li>
<li>All the way down to the mundane: “We are struggling to find the right cables to have in meeting rooms. You must go to lots of offices. What seems to work best?”</li>
</ul>
<h2 id="2-reiterate-that-we-arent-special">2. Reiterate that we aren’t special</h2>
<ul>
<li>We want to use their tools in the most normal way possible.</li>
<li>They MUST tell us if we’re trying (or intending) to do things in a strange way.</li>
<li>How should we change so that we can get the most from their products and services, and so that it’s easier for them to support us in the long term?</li>
<li>On the flip side, we might be helping them develop their future roadmap, and be an early adopter/beta partner on a future service. That’s fine, providing all are knowingly entering this arrangement.</li>
</ul>
<h2 id="3-do-push-for-industry-wide-solutions-and-collaboration">3. Do push for industry wide solutions and collaboration</h2>
<ul>
<li>With GDPR, all companies would have been going through the same things. Many of the large vendors were really really poor at working with regulators to agree “This is good CRM practice”, and then informing their customers how to do it right. Instead, all customers had to reinvent the wheel.</li>
<li>Ask who is doing things like X, even outside our sector.</li>
</ul>
<h2 id="4-avoid-talking-absolute-price-but-instead-try-pursuing-the-pricing-model-itself-is-it-a-fair-model">4. Avoid talking absolute price, but instead try pursuing the pricing model itself. Is it a fair model?</h2>
<ul>
<li>Suppliers incur costs based on their time, the amount of network traffic, storage and computing they’re doing. But they frequently have a pricing model which doesn’t correlate with these underlying drivers.</li>
<li>What this means is that when we implement their service, we may make really bad design decisions so that we can avoid cost oddities.</li>
<li>For example, if the product incurs a cost per user, but we need lots of people to have access each for only an hour per week, then we may end up sharing user-accounts. That’s not best security practice, probably reduces the utility of the software, but avoids a 10x licensing fee. Highlight these challenges to vendors and negotiate. Usually they’re open to adapting.</li>
</ul>
<h2 id="5-talk-about-table-stakes-the-absolute-basics-that-they-should-be-achieving-without-charging-extra">5. Talk about table stakes. The absolute basics that they should be achieving without charging extra.</h2>
<ul>
<li>Be utterly blunt if they’re failing to do basics/charging extra for things they shouldn’t be.</li>
<li>For example, in early 2019, it’s unacceptable to be charging customers of a SaaS product for SSL. Equally, multi-factor authentication is just part of the way of doing business.</li>
</ul>
<h2 id="he-also-had-some-other-tips">He also had some other tips</h2>
<p>These were Rob’s five top tips, but he did have some suggestions for other things the meetings could be useful for:</p>
<ul>
<li>Requesting executive level briefing – if you or other or other senior folk are new to a company who is using an existing vendor, do insist on FREE exec level briefing on the product (note that this isn’t how-to, but more a clear walk through of its capabilities and benefits), and be wary if the supplier isn’t prepared to properly explain to a tech lead or exec the value and results that the product can achieve. This is especially important when you’re inheriting an existing contract and meeting a vendor for the first time.</li>
<li>Going specific on long term things and macro concepts, e.g. “How do you see other publishers manage metadata?”</li>
<li>Seeking their suggestions on how should we be structured to make the most of the service/software, e.g. “What’s a good approach for rolling out the service? For training? For improving?”</li>
<li>Using the vendor to help find other vendors, e.g. “Who are your competitors, and why does your product remain different?”. This is also a useful way to rapidly understand what a new vendor is offering if you’re not sure.</li>
</ul>
<h2 id="and-one-to-avoid">And one to avoid:</h2>
<ul>
<li>Avoid going into details of the service unless there’s been an operational incident OR their support is letting our team down.</li>
<li>In that case, be specific about where you need support, e.g. “We’ve bought your SaaS offering so that we don’t need to manage it, yet our team seem to be spending a lot of time chasing [some problem]. Are we using it wrongly? Or is it unreliable? How are you going to restore our ability to use this product?”</li>
</ul>
Finding the next level tech job2019-01-16T00:00:00+00:00https://www.annashipman.co.uk/jfdi/finding-next-level-job.html<p>Last April, <a href="https://twitter.com/annashipman/status/966644775708479489">I joined the Financial Times as Technical Director for FT.com</a>. This has been a brilliant move. The job search was different from any of my previous job searches, as it involved an explicit step into technical leadership, and here I share how I did it and some things I learned.</p>
<h2 id="this-was-very-different-from-previous-job-changes">This was very different from previous job changes</h2>
<p>I joined my previous organisation, the <a href="https://gds.blog.gov.uk/about/">Government Digital Service</a> in 2012. At that point, I was a developer and I was looking for a role as a developer.</p>
<p>While at GDS I attained my long-held ambition to become a <a href="/jfdi/what-is-a-technical-architect.html">technical architect</a>, ultimately leading a project to build a <a href="https://www.youtube.com/watch?v=OLOaq-Xf5zU">PaaS for government</a>.</p>
<p>In 2016, I moved into a new, exciting role as the <a href="https://governmenttechnology.blog.gov.uk/2016/11/18/welcome-to-our-new-open-source-lead/">Open Source Lead</a>. This role involved influencing rather than delivery, which meant I could have a huge impact, but I had no team and wasn’t delivering a product.</p>
<p>After a year, I had <a href="/jfdi/a-year-in-the-life-os-lead.html">achieved what I set out to do in that role</a>, and knew I was ready for a new job, but unlike any of these moves, it wasn’t obvious what my next step would be.</p>
<h2 id="first-i-had-to-work-out-what-i-actually-wanted-to-do">First, I had to work out what I actually wanted to do</h2>
<p>I knew that I wanted a job that married my two previous roles – something that used my influencing skills and had a large impact, as in my Open Source Lead role, with the team and delivery of my Technical Architect role.</p>
<p>So I was looking for some kind of technical leadership. But within that, there is a lot of variation. For example, tech leadership can be just leading/managing technical people, or it can be just setting vision and direction for a technical product, or it can be a combination of both.</p>
<p>So it wasn’t immediately clear what kind of role I’d be looking for next.</p>
<h2 id="inventorying-my-skills">Inventorying my skills</h2>
<p>I was at an advantage because while I felt ready to move on, I was not obliged to rush. I still had a job and plenty of interesting things to do, so I could take a bit of time to make sure my next step was the right one.</p>
<p>The first important step was to work out what I was good at. This is something worth doing because although it seems like it might be obvious, I tend to focus on getting things done, and don’t always reflect on what skills it is that mean I’m succeeding (or what weaknesses mean I’m not).</p>
<p>There were four main ways I did this:</p>
<ol>
<li>I asked people directly what they valued about me. For example, a friend made an intro to her boss and I asked how she’d described me, and she reported that she’d said “more single minded than anyone else I know”. Another former boss called me “terrifyingly competent” (which I think was a compliment…).</li>
<li>I did an exercise called a Johari window to learn what colleagues thought my strengths are, <a href="/jfdi/johari-window.html">which I’ve written up here</a>.</li>
<li>As I started having interviews, I made sure to ask for feedback at every stage of the process. A good question to draw that out can be something along the lines of “Do you have any concerns about my ability to do this job? Are there any gaps I can perhaps set your mind at rest about?”</li>
<li>And of course, as I live my life by lists, I made a list, and updated it when I noticed I’d done something well or badly.</li>
</ol>
<p>Now that I’ve done this exercise it’s made me much more conscious and reflective about my strengths and weaknesses, which I think is helping me develop myself better.</p>
<h2 id="working-out-what-i-wanted-in-my-next-role">Working out what I wanted in my next role</h2>
<p>Because I wasn’t sure what my next steps were, I just started looking at job descriptions. As things happened at work, or I saw a job I was interested in (or very much wasn’t), I would make a note about what it was about it that appealed or repelled.</p>
<p>This was quite an interesting exercise, because it made me realise some things that are really important to me but it wouldn’t necessarily have occurred to me to mention to a recruiter.</p>
<p>For example, I cycle to work. This is a hugely important part of my life and I realised that when I see a job I’m interested in, the first thing I do is find out where the main location is, and then calculate the cycle time using <a href="https://citymapper.com/">Citymapper</a>. If it’s too long a bike ride, that’s it, the job is out of contention. The interesting bit was that this was an almost unconscious process I was going through – the location was completely non-negotiable, but it didn’t appear in my list of characteristics of my ideal job.</p>
<p>Apart from cycling, I came up with a list of what I was looking for. It included:</p>
<ul>
<li>Needs to be in an organisation that is doing something worthwhile (not, for example, Facebook).</li>
<li>Must be able to speak publically about the work and ideally blog.</li>
<li>Deliverable-based rather than BAU, even if those deliverables are a strategy or some guidance or blog posts. Not answering same questions week in week out with no possibility of effecting change. I want to be able to make things better.</li>
<li>Interesting, challenging problems to solve.</li>
<li>Must be a culture where you are allowed to take breaks, e.g. a lunch break, leaving on time, otherwise I will not do my best work.</li>
<li>They must care about the user and do user research.</li>
<li>Not a person manager role, i.e. not head of development unless that is a technology leading role. I don’t mind managing people, but I don’t want that to be my entire job.</li>
<li>Possibly CTO or, in a larger company, a role at the level below with possibility of advancement.</li>
<li>Possibly a person who sits between the business and technology team; influencing both.</li>
<li>Good balance of women and black and ethnic minorities – I didn’t want to be the diversity hire.</li>
<li>And ideally, there would be secure bike parking and fitness groups, e.g. a running club, or even something like my brilliant GDS colleague <a href="https://twitter.com/zilnhoj">John Byrne</a>’s circuit training</li>
</ul>
<p>Once I had this list, it was much easier to ask the right questions of recruiters and understand how well a job advert fit what I was looking for.</p>
<h2 id="parsing-job-titles-is-difficult">Parsing job titles is difficult</h2>
<p>Unfortunately, identifying the kind of work I wanted to do didn’t mean I could then identify the job from the job title.</p>
<p>“Engineering Manager”, “Engineering Director” and “Technical Director” could all be either entirely people-focused, entirely tech focused, or a mix. Even “CTO” varies hugely from company to company. It’s not just that in a small company you’ll be hands-on whereas in a large one you’ll be many levels away from the code. In some companies, the CTO is actually in sales, or otherwise outward facing. (This is an <a href="http://www.brixtonspa.com/Career/The_Role_of_the_CTO_4Models.pdf">interesting paper on the different kinds of CTO roles</a>.)</p>
<p>I think this will be the case for the next few steps in my career; I’ll need to know more about what the actual work is. A recruiter with whom you have a good relationship and knows what you are looking for is useful here, as they will be able to parse the job titles for you.</p>
<h2 id="i-had-24-hours-of-chats">I had 24 hours of chats</h2>
<p>As soon as I started thinking about moving on, I reached out to people for advice, and by the time I’d found my next role I had had chats with 20 different people, including former colleagues, contacts and people I’d worked for.</p>
<p>Some of the excellent advice I got from the people I talked to included:</p>
<ul>
<li>Discuss salary and working hours later in the process – first paint a picture of what you are selling.</li>
<li>Apply for some jobs out of your reach, and then when you don’t get them ask what appealed about you, and why you didn’t get the job (and it’s also possible that you might actually get the job!)</li>
<li>A good recruiter can act as an agent for you, telling you what skills you need to pick up in your current role so that they can sell you again in 2-3 years.</li>
<li>Someone asked me where I wanted to be in 5-10 years, which is an excellent question to focus the mind.</li>
<li>Think about how you would introduce yourself = “this is the kind of person I am”.</li>
<li>And someone who noted that I was taking time over choosing my next step (more on that in a moment) reminded me not to spend too long thinking before taking action by asking me “what is the risk of getting it wrong?”</li>
</ul>
<h2 id="if-you-do-nothing-else-definitely-have-some-chats">If you do nothing else, definitely have some chats</h2>
<p>Many of the chats led to introductions to other people for more chats, and many of those resolved into an actual job that I could potentially apply for.</p>
<p>This meant some of the chats were with potential employers, almost a pre-interview stage, and these were also very useful. It was a lot of effort but it was good practice. The interview process is about learning about the company as well as them interviewing you, and I made some good contacts with people in companies that I never entered into a formal process with.</p>
<p>If you are looking for a new job and you only follow one suggestion from this, it would be do reach out to people for chats. Be explicit that you are looking for a new role. It took a lot of time, but it was the most fruitful part of this process.</p>
<h2 id="i-had-21-hours-of-interviews">I had 21 hours of interviews</h2>
<p>In the end, I interviewed with four companies, which involved 21 hours of interviews with 31 different people. At this level, application processes can be very long.</p>
<p>It’s worth mentioning that I found out about all four of those jobs through chats. The jobs were all advertised, but that wasn’t how I found out about them.</p>
<p>I withdrew from one partway through the process (“Engineering manager”) because having been through two interview rounds, I felt that the role would not be challenging enough for me. I was unsuccessful in another one (“CTO”), having made it to the final round, but it was a very useful experience and I made some good contacts.</p>
<p>And I was ultimately offered two, “Engineering Director” and “Technical Director”. Both reported to the CTO, both had opportunities for development and both met all I wanted in my next role. Ultimately, I chose the Financial Times because I really believe in the mission, but it was a tough choice.</p>
<h2 id="this-all-took-a-lot-of-time-but-it-doesnt-need-to">This all took a lot of time, but it doesn’t need to</h2>
<p>I said earlier that I was at an advantage in that I didn’t have to rush, and if I’d moved when I started looking (September) then I would have missed the amazing opportunity that my current job is (came up in the first week of January) but that aside, it’s not clear it was totally an advantage; if I’d had more of an impetus, it may not have needed to take so long.</p>
<p>As my <a href="https://twitter.com/proustian/">excellent friend</a> pointed out, the risks of taking an imperfect next step might not have been so high.</p>
<p>I’m really happy with where I landed, and I learned a lot from the process, but if you plan to follow what I did, you don’t necessarily need to allow four months.</p>
<p>However, the right technical leadership role can take some time to come up – unlike developer roles, there isn’t always one out there somewhere. If you do find it takes time, you can always use that time to think about particular things you can achieve in your current role that will help you with what you are looking for.</p>
<h2 id="good-questions-i-asked">Good questions I asked</h2>
<p>Here are some of the questions that I asked in chats or interviews that got me really useful information about the job.</p>
<ul>
<li>How are products/new features committed to?</li>
<li>How are tech decision made/sold?</li>
<li>What does success look like in this role?</li>
<li>What’s the biggest challenge facing the organisation?</li>
<li>What are you hoping to improve/how is that measured?</li>
</ul>
<p>A good question to ask at the end of an interview is the one I mentioned above; “Do you have any reservations about my suitability for this role?” One person gave me ten minutes worth of very useful, actionable feedback after I asked this. So much so that I suspected I wasn’t going to get the job, but he put me through to the next round.</p>
<p>And a bonus, not necessarily for interviews but a really good question someone asked in a presentation once and I share here: “You’ve said how you see it working, how do you see this failing?”</p>
<h2 id="good-questions-i-got-asked">Good questions I got asked</h2>
<p>I got asked a lot of very good and/or difficult questions (though almost no <a href="https://www.annashipman.co.uk/jfdi/interviewing-fairly.html">competency-based questions</a>). One advantage of doing a lot of interviews at once is that you get into the swing of answering questions like this. But it’s always worth practising, so here are some of the ones I was asked.</p>
<ul>
<li>What does good leadership look like? Followed by: How do you make sure people who report to you get that?</li>
<li>Describe your leadership style.</li>
<li>Looking back over your career, what has been your biggest failure?</li>
<li>What are you most proud of?</li>
<li>What would your direct reports say is your best quality and what would they wish you would change?</li>
<li>How do you measure success of your work? Give some examples.</li>
<li>What would you say your best quality is – what’s your superpower? Followed by: How do you pass this on to your reports?</li>
<li>Can you give me an example of a time that a project was running behind, how you knew this and what you did?</li>
</ul>
<h2 id="bad-questions-i-got-asked">Bad questions I got asked</h2>
<p>I also got asked some bad questions. A red flag was when people only wanted to talk about my previous job but one, i.e. my job as Technical Architect for GOV.UK PaaS.</p>
<p>My year as Open Source Lead was great, and I developed a lot of very useful skills, particularly around influencing, and identifying the highest impact areas to focus on. However, a few potential companies didn’t find my most recent experience relevant and only wanted to ask me very specific questions about team management, like ‘what dashboards do you make sure your team has up?’</p>
<p>Yes, I can manage a team, but that is not my main value to your organisation. When interviewers focused on this, it made it clear to me that this wasn’t the role I was looking for.</p>
<h2 id="the-hardest-question-i-got-asked">The hardest question I got asked</h2>
<p>I had three interviews for my job at the Financial Times (with a total of 8 people). The last one was with the CTO John Kundert, now my boss, and the very first question was the hardest question I was asked in the all of the 21 hours of interviews in the previous few months:</p>
<ul>
<li>What are your main values, and how do they influence your work?</li>
</ul>
<p>As with some of the questions above, all of the thinking I’d done about my strengths and values, plus the practice I’d recently had in inteviewing elsewhere, made this something I could answer both honestly and usefully.</p>
<p>It was a very difficult question, especially as the opener, but it also demonstrated that this was an organisation, and a manager, who would challenge me to be my best.</p>
<h2 id="how-it-worked-out">How it worked out</h2>
<p>I’ve been at the FT for eight months now as Technical Director and it’s everything I hoped for. It makes use of some of the skills I already have, for example influence, clarity of thought, bias for structure and execution; while giving me the opportunity to develop some others, for example stakeholder management, managing a budget.</p>
<p>I’m sure that at least one of the other jobs in contention would also have been great, and approaching this process in good faith had unintended benefits. For example, it was a very good way to network. I received a free invitation to a tech conference from one job that didn’t work out, and another one invited to be on a tech advisory board. Tech is a small world, and I’m sure I will be working with many of the people I met later in my career.</p>
<p>And I’ve even set up <a href="https://twitter.com/annashipman/status/1034409955011567621">my own circuits club</a>!</p>
What I learned in six years at GDS2018-12-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/what-i-learned-in-six-years-at-gds.html<p>When I joined the Government Digital Service in April 2012, <a href="https://www.gov.uk/">GOV.UK</a> was just going into public beta. GDS was a completely new organisation, part of the Cabinet Office, with a mission to stop wasting government money on over-complicated and underperforming big IT projects and instead deliver simple, useful services for the public.</p>
<p>Lots of people who were experts in their fields were drawn in by this inspiring mission, and I learned loads from working with some true leaders. Here are three of the main things I learned.</p>
<h2>1. What is the user need?</h2>
<p> The main discipline I learned from my time at GDS was to always ask ‘what is the user need?’ It’s very easy to build something that seems like a good idea, but until you’ve identified what problem you are solving for the user, you can’t be sure that you are building something that is going to help solve an actual problem.</p>
<p>A really good example of this is GOV.UK Notify. This service was originally conceived of as a status tracker; a “where’s my stuff” for government services. For example, if you apply for a passport online, it can take up to six weeks to arrive. After a few weeks, you might feel anxious and phone the Home Office to ask what’s happening. The idea of the status tracker was to allow you to get this information online, saving your time and saving government money on call centres.</p>
<p>The project started, as all GDS projects do, with a <a href="https://www.gov.uk/service-manual/agile-delivery/how-the-discovery-phase-works">discovery</a>. The main purpose of a discovery is to identify the users’ needs. At the end of this discovery, the team realised that a status tracker wasn’t the way to address the problem. As they wrote in <a href="https://gds.blog.gov.uk/2015/10/05/status-tracking-making-it-easy-to-keep-users-informed/">this blog post</a>: </p>
<div class="quote">
<p>Status tracking tools are often just ‘channel shift’ for anxiety. They solve the symptom and not the problem. They do make it more convenient for people to reduce their anxiety, but they still require them to get anxious enough to request an update in the first place.</p>
</div>
<p>What would actually address the user need would be to give you the information before you get anxious about where your passport is. For example, when your application is received, email you to let you know when to expect it, and perhaps text you at various points in the process to let you know how it’s going. So instead of a status tracker, the team built <a href="https://www.notifications.service.gov.uk/">GOV.UK Notify</a>, to make it easy for government services to incorporate text, email and even letter notifications into their processes.</p>
<h3>Making sure you know your user</h3>
<p>At GDS user needs were taken very seriously. We had a user research lab on site and everyone was required to spend two hours observing user research every six weeks. Ideally you’d observe users working with things you’d built, but even if they weren’t, it was an incredibly valuable experience, and something you should seek out if you are able to.</p>
<p>Even if we think we understand our users very well, it is very enlightening to see how users actually use your stuff. Partly because in technology we tend to be power users and the average user doesn’t use technology the same way we do. But even if you are building things for other developers, someone who is unfamiliar with it will interact with it in a way that may be very different to what you have envisaged.</p>
<h3>User needs is not just about building things</h3>
<p>Asking the question “what is the user need?” really helps focus on why you are doing what you are doing. It keeps things on track, and helps the team think about what the actual desired end goal is (and should be). </p>
<p>Thinking about user needs has helped me with lots of things, not just building services. For example, you are raising a pull request. What’s the user need? The reviewer needs to be able to easily understand what the change you are proposing is, why you are proposing that change and any areas you need particular help on with the review. </p>
<p>Or you are writing an email to a colleague. What’s the user need? What are you hoping the reader will learn, understand or do as a result of your email?</p>
<h2>2. Make things open: it makes things better</h2>
<p>The second important thing I learned at GDS was ‘make things open: it makes things better’. This works on many levels: being open about your strategy, blogging about what you are doing and what you’ve learned (including mistakes), and – the part that I got most involved in – coding in the open.</p>
<h3>Talking about your work helps clarify it</h3>
<p>One thing we did really well at GDS was blogging – a lot – about what we were working on. Blogging about what you are working on is is really valuable for the writer because it forces you to think logically about what you are doing in order to tell a good story. If you are blogging about upcoming work, it makes you think clearly about why you’re doing it; and it also means that people can comment on the blog post. Often people had really useful suggestions or clarifying questions.</p>
<p>It’s also really valuable to blog about what you’ve learned, especially if you’ve made a mistake. It makes sure you’ve learned the lesson and helps others avoid making the same mistakes. As well as blogging about lessons learned, GOV.UK also publishes <a href="https://insidegovuk.blog.gov.uk/category/incident-reports/">incident reports</a> when there is an outage or service degradation. Being open about things like this really engenders an atmosphere of trust and safe learning; which helps make things better.</p>
<h3>Coding in the open has a lot of benefits</h3>
<p>In my last year at GDS I was the Open Source Lead, and one of the things I focused on was the <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">requirement that all new government source code should be open</a>. From the start, GDS coded in the open (the GitHub organisation still has the non-intuitive name <a href="https://github.com/alphagov/">alphagov</a>, because it was created by the team doing the original Alpha of GOV.UK, before GDS was even formed).</p>
<p>When I first joined GDS I was a little nervous about the fact that anyone could see my code. I worried about people seeing my mistakes, or receiving critical code reviews. (Setting people’s mind at rest about these things is why it’s crucial to have good standards around communication and positive behaviour - even a critical code review should be considerately given). </p>
<p>But I quickly realised there were huge advantages to coding in the open. In the same way as blogging your decisions makes you think carefully about whether they are good ones and what evidence you have, the fact that anyone in the world could see your code (even if, in practice, they probably won’t be looking) makes everyone raise their game slightly. The very fact that you know it’s open, makes you make it a bit better.</p>
<p>It <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">helps with lots of other things as well</a>, for example it makes it easier to collaborate with people and share your work. And now that I’ve left GDS, it’s so useful to be able to look back at code I worked on to remember how things worked.</p>
<h3>Share what you learn</h3>
<p>It’s sometimes hard to know where to start with being open about things, but it gets easier and becomes more natural as you practice. It helps you clarify your thoughts and follow through on what you’ve decided to do. Working at GDS when this was a very important principle really helped me learn how to do this well.</p>
<h2>3. Do the hard work to make it simple (tech edition)</h2>
<p>‘Start with user needs’ and ‘Make things open: it makes things better’ are two of the excellent <a href="https://www.gov.uk/guidance/government-design-principles">government design principles</a>. They are all good, but the third thing that I want to talk about is number 4: ‘Do the hard work to make it simple’, and specifically, how this manifests itself in the way we build technology.</p>
<p>At GDS, we worked very hard to do the hard work to make the code, systems and technology we built simple for those who came after us. For example, writing good commit messages is taken very seriously. There is <a href="https://github.com/alphagov/styleguides/blob/master/git.md">commit message guidance</a>, and it was not unusual for a pull request review to ask for a commit message to be rewritten to make a commit message clearer.</p>
<p>We worked very hard on <a href="https://www.annashipman.co.uk/jfdi/good-pull-requests.html">making pull requests good</a>, keeping the reviewer in mind and making it clear to the user how best to review it.</p>
<p>Reviewing others’ pull requests is the highest priority so that no-one is blocked, and teams have screens showing the status of open pull requests (using <a href="https://github.com/alphagov/fourth-wall">fourth wall</a>) and we even had a ‘<a href="https://gdstechnology.blog.gov.uk/2015/09/24/reminding-developers-about-code-reviews/">pull request seal</a>’, a bot that publishes pull requests to Slack and gets angry if they are uncommented on for more than two days.</p>
<h3>Making it easier for developers to support the site</h3>
<p>Another example of doing the hard work to make it simple was the opsmanual. I spent two years on the web operations team on GOV.UK, and one of the things I loved about that team was the huge efforts everyone went to to be open and inclusive to developers.</p>
<p>The team had some people who were really expert in web ops, but they were all incredibly helpful when bringing me on board as a developer with no previous experience of web ops, and also patiently explaining things whenever other devs in similar positions came with questions. </p>
<p>The main artefact of this was the opsmanual, which contained write-ups of how to do lots of things. One of the best things was that every alert that might lead to someone being woken up in the middle of the night had a link to documentation on the opsmanual which detailed what the alert meant and some suggested actions that could be taken to address it.</p>
<p>This was important because most of the devs on GOV.UK were on the on-call rota, so if they were woken at 3am by an alert they’d never seen before, the opsmanual information might give them everything they needed to solve it, without the years of web ops training and the deep familiarity with the GOV.UK infrastructure that came with working on it every day.</p>
<h3>Developers are users too</h3>
<p>Doing the hard work to make it simple means that users can do what they need to do, and this applies even when the users are your developer peers. At GDS I really learned how to focus on simplicity for the user, and how much better this makes things work.</p>
<h2>These three principles help us make great things</h2>
<p>I learned so much more in my six years at GDS. For example, the civil service has a <a href="https://www.annashipman.co.uk/jfdi/interviewing-fairly.html">very fair way of interviewing</a>. I learned about the <a href="https://russelldavies.typepad.com/planning/2015/06/doing-the-hard-work-to-make-it-clear.html">importance of good comms</a>, <a href="https://dan.carley.co/blog/2014/05/21/working-late-responsibly/">working late, responsibly</a> and the value of <a href="https://contentdesign.london/book/">content design</a>.</p>
<p>And the real heart of what I learned, the guiding principles that help us deliver great products, is encapsulated by the three things I’ve talked about here: think about the user need, make things open, and do the hard work to make it simple.</p>
<p><em>This post originally appeared on <a href="https://24ways.org/2018/what-i-learned-in-six-years-at-gds/">24ways</a></em>.</p>
How to interview job applicants fairly2018-11-22T00:00:00+00:00https://www.annashipman.co.uk/jfdi/interviewing-fairly.html<p>Interviews are not a great way to find out how good someone will actually be at the job. They can also make it easy for your unconscious biases to have too much influence. This has led some to suggest that interviews are broken and we need to find other ways; but it is possible to do interviews in a much fairer and more informative way.</p>
<h2 id="the-best-predictor-of-future-performance-is-past-performance">The best predictor of future performance is past performance</h2>
<p>Let’s say you want someone who is good at setting vision and direction for a large team. A question like “How would you set vision and direction for a large team?” is only going to give you hypothetical answers.</p>
<p>All you will learn is how they talk about doing it, how much they’ve read around the topic of vision-setting, and also how much what they say chimes with what you believe. But we know that planning to do something a certain way and actually doing it that way are two very different things. They might say all the right things, but then not be able to follow through.</p>
<p>A better question would be “Tell us about a time that you set vision for a team”, because that will get them to talk about what their actual past behaviour was, which is a much better indicator of what their future behaviour will be. It also gives you an opportunity to ask questions to find out more what their approach was, like “How did you do it?”, “What was the result?”, “What didn’t go so well?” None of those questions are hypothetical, so you’re not asking the candidate to make something up that you want to hear, you are asking them what they did, and how they thought about it.</p>
<p>Note that it doesn’t have to be the exact same thing, because chances are this job will be a step up in one way or another. You need to work out what the important part is, and ask about that. So my question above was about setting vision for a team, not for a large team, because the important thing I wanted to know about was vision-setting.</p>
<h2 id="make-the-questions-relevant-to-the-main-skills-in-the-job-they-will-be-doing">Make the questions relevant to the main skills in the job they will be doing</h2>
<p>Interviewing like this is something the civil service does very well. When advertising a job, the hiring manager works out what competencies the job will involve.</p>
<p>Let’s say you are hiring a tech lead. What should this person be able to do? You might come up with some things like set technical direction, communicate effectively with other teams, communicate effectively with the business, unblock team members.</p>
<p>You may come up with a very long list. Identify the four or five that are essential to the role, and then devise questions around those skills. “Can you give us an example of a time that you unblocked a team member?” for example. Or, “Tell us about a time that there was a breakdown in communication between engineers and the business. What did you do to improve the situation?”</p>
<p>Spending time thinking about what exactly you want from the candidate is time consuming, but nowhere near as time consuming as hiring the wrong person.</p>
<h2 id="make-sure-the-questions-have-clear-criteria-that-you-can-judge-them-on">Make sure the questions have clear criteria that you can judge them on</h2>
<p>As well as being relevant to the role and asking about past performance, you need to make sure the questions can be answered in a way that you can aim to objectively judge the answer. For example, with the vision-setting question, you’d be looking for things like whether they’d considered a diverse audience, whether and how they measured the impact, etc.</p>
<p>If you ask a vague question like “Have you worked in an Agile environment? What did you find positive and negative about it?”, what criteria can you judge their answer on, other than whether that sounds like the kind of thing you’d agree with?</p>
<h2 id="ask-all-candidates-the-same-questions">Ask all candidates the same questions</h2>
<p>Once you’ve identified the most important areas, write out the questions and ask all candidates the same ones. This may feel unnatural, as you will be following a script, but it’s the only way to make sure that you aren’t just making biased assumptions and are actually comparing the information each candidate gives you with each other candidate.</p>
<p>I recommend you come up with around 5 or 6 questions. Once you’ve allowed time for introducing the panel, explaining the interview, and 5-10 minutes at the end for answering their questions, then six is the most you can reasonably cover in an hour.</p>
<p>Some people do go into a lot of detail, so be prepared to cut people off if necessary – you want to give them the best chance possible to demonstrate the full range of skills.</p>
<h2 id="encourage-them-to-give-you-all-the-information-you-need-for-the-answer">Encourage them to give you all the information you need for the answer</h2>
<p>You want to know as much as possible about the extent and effectiveness of what they did, so you can understand whether they have the skills you are looking for.</p>
<p>A useful model to think of is the context/action/result model (a useful mnemonic is CAR). If they don’t tell you all that as part of their answer, you can prompt them.</p>
<p>For example, using the question about unblocking team members. Context: Why were the team members blocked? Action: What did you do to unblock them? Result: What was the outcome?</p>
<p>This will give them a better opportunity to make sure you have the information you need. Remember, this is about giving them the opportunity to give you the most useful information to help you make a decision about whether they are right for the role.</p>
<p>(For candidates, I’ve got a bit more advice about how to structure your answers and think of examples in <a href="https://gdstechnology.blog.gov.uk/2013/12/24/applying-for-a-job-at-gds/">this blog post for the Government Digital Service</a> or read this <a href="https://twitter.com/edinbeth/status/1072518477196464130">excellent Twitter thread by Beth Fraser</a>.)</p>
<h2 id="score-the-questions-immediately">Score the questions immediately</h2>
<p>In order to make the most of the process, you need to give a score for each question as soon as you can. Ideally immediately after the interview, so that you can remember what has been discussed. This helps you focus on how they performed against each of the important skills, rather than being influenced by what you thought of them personally, or your general sense of whether they’d be good (which will be informed by your unconscious biases).</p>
<p>This also helps address the <a href="https://dataworks-ed.com/blog/2014/08/the-primacyrecency-effect/">primacy/recency effect</a>, where you will think more highly of them if they answered the first and last question well, or if they were the first or last candidate.</p>
<p>At GDS we used a score of 0-3.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0. They demonstrated no evidence of this skill.
(e.g. if the question is about Perl, they cannot write it at all).
1. They showed that with support, they can master this skill
(They made a few mistakes, they'll get there)
2. They have definitely demonstrated this skill
(They can start writing Perl on day one)
3. They've exceeded the skill level required for this role.
(They are Larry Wall*).
</code></pre></div></div>
<p><a href="https://en.wikipedia.org/wiki/Larry_Wall">*</a></p>
<h2 id="have-more-than-one-person-in-each-interview">Have more than one person in each interview</h2>
<p>To avoid biases, you need at least two perspectives and ideally three, and the interviewers should be different from each other to help with that: for example, from different disciplines, or different levels (e.g. a junior and a senior), or different ethnic backgrounds, etc.</p>
<p>Each person should make notes during the interview, to allow comparison and feedback later, and then each give a score for each question on your own before discussing as a panel, to guard against your score being influenced by what the others think.</p>
<h2 id="set-people-up-to-succeed">Set people up to succeed</h2>
<p>Interviews are not meant to be a stressful test, or have trick questions. You want to get the right person for the job, not outwit anyone. So the whole process should be clear and transparent to the candidate.</p>
<p>In the job advert, spell out skills you are looking for. Let them know when you are inviting them for interview that they can prepare by thinking about examples of times they’ve demonstrated those skills, and perhaps tell them about the context/action/result model.</p>
<p>This means that people will be set up to do their best and you’ll be able to really give them a chance to shine. And it will also mean that you can give clear feedback on what areas to work on for unsuccessful candidates.</p>
<h2 id="further-reading">Further reading</h2>
<p>If you want to read more about this, here is an <a href="https://www.nytimes.com/2017/04/08/opinion/sunday/the-utter-uselessness-of-job-interviews.html">article about why interviews are useless</a>. But, as a throwaway question about what can be done at the end, it says “One option is to structure interviews so that all candidates receive the same questions, a procedure that has been shown to make interviews more reliable and modestly more predictive of job success. Alternatively, you can use interviews to test job-related skills, rather than idly chatting or asking personal questions.”</p>
<p>How to do exactly that is what I’ve described here.</p>
High output management2018-11-05T00:00:00+00:00https://www.annashipman.co.uk/jfdi/high-output-management.html<p>Earlier this year I read <a href="https://www.amazon.co.uk/High-Output-Management-Andrew-Grove/dp/0679762884/">High Output Management</a>. This is one of the best books I’ve read on leadership, and I really recommend you read it. Some of the things I took from it are here.</p>
<p>It’s really excellent on what the important things are when running a large team or business. Andy Grove was co-founder and CEO of Intel, so it’s written by someone who was actually doing the work at the time, unlike lots of other management books.</p>
<p>A lot of it really resonated with me because it reflected some of my own views, but it also changed my perspective on a few things; particularly the value of 1-1s and on how much time is reasonable to be spending in meetings.</p>
<h2 id="the-art-of-management-is-selecting-which-activities-will-provide-most-leverage">The art of management is selecting which activities will provide most leverage</h2>
<p>“Like a housewife’s, a manager’s work is never done.”</p>
<p>There is always more you could or should be doing, so the skill in being a manager is in shifting your attention to the activities which will have the most impact on increasing the output of your organisation. In other words, working out which activities are the ones where your leverage will be greatest.</p>
<h2 id="the-highest-leverage-activity-is-developing-your-reports">The highest leverage activity is developing your reports</h2>
<p>This is the highest impact thing you can do, because the more senior your reports are, the more they can take on big pieces of work that will free you up.</p>
<p>This makes 1:1s with your reports one of the most important things you can spend your time on. He suggests the agenda for the 1:1 should be set by the report, but also says you should cover current problems, plans and what’s worrying them/what they see as future problems.</p>
<p>Another suggestion he makes is that being part of a peer group is a way to increase your leverage by affecting the work of all. This one was counter-intuitive to me, but after six months of being involved in the Technical Leadership Group at the Financial Times, I can really see what he means by this.</p>
<h2 id="your-output-is-the-output-of-your-team">Your output is the output of your team</h2>
<p>The output of a manager is a result of a group under her supervision or management, or as my excellent boss <a href="https://twitter.com/caitoriordan/">Cait O’Riordan</a> puts it, “you’re nothing without your team”.</p>
<p>He gives the example of a fire station. You have to shape an energetic team that can deal with whatever comes up.</p>
<h2 id="delegating-is-extremely-high-leverage">Delegating is extremely high leverage</h2>
<p>He points out that the “delegator” and the “delegatee” need to have a common set of operational ideas about how to go about solving problems, otherwise the delegatee can only become an effective proxy if given explicit instructions. Like with micro-managing, this really isn’t effective in multiplying your impact, or, as he puts it “meddling produces low managerial leverage”.</p>
<p>So a manager must communicate her objectives, priorities and preferred approaches, and you need a shared set of values to effectively delegate. He also says that
following the corporate culture makes it easy for people to make decisions.</p>
<p>If that common base of information and similar ways of doing and handling things is created, this can exert enormous leverage.</p>
<h2 id="he-has-a-lot-of-very-useful-advice-about-how-to-delegate">He has a lot of very useful advice about how to delegate</h2>
<p>Something that is a small task to you may be a much larger proposition to the person you delegate it to. “A senior manager’s tactics might be the next manager down’s strategy.”</p>
<p>He also has suggestions for how to effectively delegate; following the principles of quality assurance in manufacturing that he introduced early in the book, e.g. review rough drafts of reports and sample output (more often if they are unfamiliar with the task, and then ease off). The approach depends on the “task-relevant maturity” of the employee. Operational reviews and presentations are good source of motivation as people will want to make a good impression.</p>
<p>“Monitoring” – giving people objectives and checking in – is on paper a manager’s most productive approach but we have to work our way up to it, and if things suddenly change we may have to revert quickly to much closer management of the task. But we are biased against that, so sometimes we don’t take it up where necessary until too late. We need to think about the most effective way to manage, not be guided by what we feel is good/bad.</p>
<h2 id="meetings-are-the-medium-of-managerial-work">Meetings are the medium of managerial work</h2>
<p>This one is really interesting, and he calls out that he disagrees with <a href="https://www.waterstones.com/book/the-effective-executive/peter-drucker/9780750685078">Peter Drucker</a> on this. Drucker says that spending more than 25% of your time in meetings shows that “there is time-wasting malorganization”. Certainly, coming from an engineering background, I tend to feel the fewer the meetings, the better.</p>
<p>But Andy Grove expressly disagrees with this. He points out that, as above, a big part of your role is to share know-how, and “impart a sense of the preferred method of handling things” to your reports. This can only really be done in person, and therefore, during meetings. “Thus I will assert again that a meeting is nothing less than the <em>medium</em> through which managerial work is performed”. So we shouldn’t be fighting their existence, but just making sure that we are using the time as efficiently as possible.</p>
<p>This has really changed my perspective on a lot of the meetings I have.</p>
<h2 id="be-available">Be available</h2>
<p>The most important thing in making good decisions is getting information; and usually this is better if verbal and quick, rather than written reports. It’s useful to go round and chat to people. Reports are mostly useful because they force people to clarify their thoughts, so the writing is more important than the reading.</p>
<p>So make sure you schedule slack into your calendar. But you should also have a list of non-urgent things you can do that will increase productivity – otherwise you will use your free time to meddle in your reports’ work, which will slow things down.</p>
<h2 id="how-to-deal-with-interruptions">How to deal with interruptions</h2>
<p>Hiding physically in order to get work done is not great, because people do have genuine questions or issues and you’ll slow things down.</p>
<p>One approach is to standardise responses, which can then also help with delegation.</p>
<p>Another is batching – if you have regular catch-ups people can’t complain about being asked to batch and bring to you at scheduled times. You could also have office hours.</p>
<p>“To make something regular that was once irregular is a fundamental production principle and that’s how you should try to handle the interruptions that plague you”.</p>
<h2 id="performance-review-is-really-important">Performance review is really important</h2>
<p>Don’t ask your reports to review themselves first. You should be paying attention. “Reviewing the performance of subordinates is a formal act of leadership.”</p>
<p>He scrutinises a selection of performance reviews given by other managers to their reports throughout the year, with as much visibility as possible, to make it clear it’s the most important kind of task-relevant feedback we can give our reports.</p>
<p>When delivering a performance review make sure your report receives the message. Don’t say too many things in a performance review as they may not take it in – focus on the most important things. He also has some good, practical advice on how to prepare for and run a performance review.</p>
<p>The performance rating of a manager cannot be higher than what you would give the organisation under her control.</p>
<p>If there is a problem and they don’t agree with your assessment but commit to your proposed solution, that’s fine. To make things work people don’t need to side with you, they only need to commit to pursue a course of action that’s been decided on. He also has some practical advice on how to get that commitment.</p>
<h2 id="when-reviewing-a-star-performer-make-sure-to-focus-on-how-they-can-improve-their-performance">When reviewing a star performer make sure to focus on how they can improve their performance</h2>
<p>He points out that often the reviews of star performers focus a lot on what they’ve done well, whereas reviews of poor performers focus on how they can improve, but in fact we should spend more time on focusing on how star performers can improve. It’s a high leverage activity because it will have a big impact on group output.</p>
<h2 id="develop-culture-and-values-by-what-you-do-as-well-as-what-you-say">Develop culture and values by what you do as well as what you say</h2>
<p>He had two important examples of this:</p>
<ol>
<li>When you choose to promote someone you are signalling your values and who you see as role models</li>
<li>If a valued employee wants to quit your top priority becomes that – not going to whatever important meeting you are on your way to. Your first reaction counts.</li>
</ol>
<h2 id="how-to-use-objectives-and-key-results">How to use objectives and key results</h2>
<p>OKRs are usually attributed to Google, but while reading this book, I realised that in fact, they originated here. Andy Grove developed Peter Drucker’s <a href="https://www.waterstones.com/book/the-effective-executive/peter-drucker/9780750685078">Management by objectives</a> into OKRs, and then John Doerr learned them at Intel and took them to Google. Reading them here was the first time that they actually made sense to me rather than feeling cargo-culted.</p>
<p>Objectives are what you need to do; key results are how you know you are on your way. The example that really made it clear to me was: your objective is to reach the airport in an hour. Key results are: pass through town A at 10 mins, B at 20 mins, C at 30 mins. If after 30 mins there is no sign of town A, you know you’ve gone off track. So they need to be clear enough that you know you’ve met them, and that you are on track.</p>
<p>He points out that the system requires judgment and common sense. Objectives are not a legal document. If the manager mechanically relies on the OKRs for the review, or the report ignores an emerging opportunity because it wasn’t one of the objectives, “then both are behaving in a petty and unprofessional fashion”.</p>
<p>And finally, a very important point: you should not have too many! “To focus on everything is to focus on nothing”.</p>
<h2 id="everyone-should-be-able-to-contribute-to-decisions-but-then-commit-even-if-they-disagree">Everyone should be able to contribute to decisions, but then commit even if they disagree</h2>
<p>The first stage of making decisions with a team needs to be free discussion. It’s very important that everyone feels they can contribute, because more junior people are likely to have a better grasp of the technology than senior managers, but senior people have experience from previous errors, etc.</p>
<p>Once a decision is made it needs to be clear, and it needs to be supported by everyone, even those who don’t agree.</p>
<p>Confidence in these discussions comes from realising it’s OK to make mistakes, so it’s important to make sure everyone understands this. That is, in this book written 35 years ago, he’s telling us about the importance of <a href="https://hbr.org/2017/08/high-performing-teams-need-psychological-safety-heres-how-to-create-it">psychological safety</a>.</p>
<h2 id="say-no-as-early-as-possible">Say no as early as possible</h2>
<p>In manufacturing, it is vitally important to reject defective material at its lowest value (i.e. when you’ve not done much work on it). Similarly with managerial work.
Say no as early as possible. Getting an unnecessary meeting called off early is the same as stopping work on something at a low value-added time.</p>
<p>Remember: saying yes to one thing means saying no to another. And we can say no in two ways – one by saying no, and one by saying yes but not actually doing it. The latter wastes time and energy.</p>
<h2 id="plan-for-the-future-not-fixing-the-present">Plan for the future, not fixing the present</h2>
<p>The final thing I wanted to note was what he says about planning. If you see a gap today, that represents a failure of planning in the past, and if you focus your energy on fixing that, you are constantly chasing after things that should already have happened. You need to instead focus on future events.</p>
<p>“Remember that as you plan you must answer the question: What do I have to do <em>today</em> to solve – or better, avoid – <em>tomorrow’s</em> problem?”</p>
<h2 id="you-should-read-the-book">You should read the book</h2>
<p>It was really hard to write this post because I made so many notes while reading it. When refreshing my memory of the book while drafting it, I wanted to make so many more, and this really just scratches the surface. It’s an excellent book and well worth your time reading it.</p>
<h2 id="three-other-books-i-would-recommend">Three other books I would recommend</h2>
<p>I’ve read lots of other good books on work-related topics, but the three that have had the most impact on me are:</p>
<h3 id="dont-make-me-think"><a href="https://www.amazon.co.uk/Dont-Make-Think-Revisited-Usability/dp/0321965515">Don’t make me think</a></h3>
<p>This is an excellent book on web usability. If you have anything at all to do with producing websites and haven’t read it already, you absolutely must.</p>
<h3 id="thinking-fast-and-slow"><a href="https://www.waterstones.com/book/thinking-fast-and-slow/daniel-kahneman/9780141033570">Thinking fast and slow</a></h3>
<p>This is a quite dense read but it’s extremely worth the effort. It explains how our minds trick us and what we can do to avoid making mistakes by biased thinking.</p>
<h3 id="slack"><a href="https://www.amazon.co.uk/Slack-Getting-Burnout-Busywork-Efficiency/dp/0767907698/">Slack</a></h3>
<p>In itself this is not an amazing book but it’s worth reading for the point about how you need some slack at work. It’s also a short read. Two important points from it have stuck with me:</p>
<ol>
<li>The example of a <a href="http://www.jeanierhoades.com/wp-content/uploads/2012/01/old-slide-puzzle.jpg">sliding tile puzzle</a>. Without the empty slot, it’s a much more efficient use of space but you can’t actually do the puzzle. This has stuck with me a really powerful metaphor as to why we need slack at work.</li>
<li>His explanation of why “people are not fungible resources”. If you have someone working half time on two teams, it’s not the case that they are 50% on each team and that adds up to 100%; it’s more like 30 or 40% on each team. The more teams you split people between, the less total time they’re able to actually function.</li>
</ol>
<p>For a better explanation of these two points alone, this is definitely worth the short time it will take you to read it.</p>
<h2 id="do-let-me-know-what-you-think">Do let me know what you think</h2>
<p>I’d be interested to hear about books that you recommend, or what you thought of these books if you’ve read them.</p>
Moving SPA to HTTPS2018-08-03T00:00:00+00:00https://www.annashipman.co.uk/jfdi/switching-spa-to-https.html<p>I had long been thinking I must move the <a href="https://www.spaconference.org/">SPA site</a> to HTTPS. However, everything is difficult when using shared hosting and I wasn’t sure how to do it. So I was delighted when <a href="https://twitter.com/YoPeeters">Johan Peeters</a> and <a href="https://twitter.com/nelisboucke">Nelis Boucké</a> proposed a session for the 2017 SPA conference called <a href="http://spaconference.org/spa2017/sessions/session703.html">Why SPA should switch to HTTPS and how easy that is</a>.</p>
<p>It was a great session. In short, they recommended three approaches:</p>
<ol>
<li>Get the hosting provider to do it. Before the session, Yo and Nelis asked me who the SPA hosting provider was (it isn’t discoverable through <code class="language-plaintext highlighter-rouge">dig</code> as it’s resellers all the way down). Once I had told them, they did a bit of investigation, and the hosting provider we use charges £180 per year for HTTPS. A bit much for a non-profit conference run by a registered charity.</li>
<li>Use <a href="https://letsencrypt.org/">Let’s Encrypt</a>. This is a free certificate authority, provided by the non-profit Internet Security Research Group whose aim is to make a secure and privacy-respecting web. The advantage of this is that you are managing it yourself (though that is also a disadvantage as it’s then up to you to make sure that you get the configuration right and keep up to date with changes). If you are using nginx or Apache, it can be a little tricky to configure, though that is addressed if you use <a href="https://caddyserver.com/docs/automatic-https">Caddy</a>. However, Let’s Encrypt won’t work for SPA, as the shared hosting does not allow us to install anything.</li>
<li>Use <a href="https://www.cloudflare.com/plans/">Cloudflare’s free plan</a>. You need to point your nameservers at Cloudflare. It means that they terminate your TLS and you are sharing a certificate with many other websites, plus the connection from your server to Cloudflare may still be unencrypted, but it’s free, easy and massively reduces the attack vectors and ability for people to sniff your network traffic. (<a href="https://www.troyhunt.com/cloudflare-ssl-and-unhealthy-security-absolutism/">This is an excellent article</a> about the security aspects of Cloudflare’s service).</li>
</ol>
<p>So it seemed like Cloudflare was the only plausible option for SPA.</p>
<h2 id="first-i-set-cloudflare-up-for-this-site">First, I set Cloudflare up for this site</h2>
<p>Prior that stage, my blog did not use HTTPS, and I didn’t see why it would need to, as all the content is <a href="https://github.com/annashipman/annashipman.github.io">open anyway</a>.</p>
<p>But Yo and Nelis made the excellent point that even static sites with non-private content should be HTTPS, because if only sites that need to be private use TLS then that leaks information just by that fact.</p>
<p>Setting Cloudflare TLS up for my site was easy. I did that in the break after the session, and it had propagated within a few hours.</p>
<p>That went so well I decided to move on to doing the SPA site shortly after lunch. This proved to be significantly more tricky.</p>
<h2 id="css-stopped-working">CSS stopped working</h2>
<p>After the nameservers had been changed to Cloudflare’s nameservers, the CSS was no longer available. The site looked like this:</p>
<p><img src="/img/no_css.png" alt="SPA homepage with no CSS" /></p>
<p>My initial response was to publish the site. Big mistake.</p>
<p><img src="/img/empty_page.png" alt="SPA programme page with no content at all" /></p>
<h2 id="frantically-trying-to-get-the-site-back-online">Frantically trying to get the site back online</h2>
<p>Before I go on, I should point out that this was in the afternoon of the last day of the conference. (Thank goodness it wasn’t the first day!) So while most presenters and attendees were at the post-conference pub drinks, I was at home frantically trying to get something back where the website had been so that when presenters went to upload their outputs from sessions there was something there for them to upload to.</p>
<p>In fact, the <code class="language-plaintext highlighter-rouge">/scripts</code> pages were fine, so they could have uploaded their outputs… if they’d known to navigate directly to the page.</p>
<h2 id="the-problem-was-with-how-we-used-the-cms">The problem was with how we used the CMS</h2>
<p>At that stage, the static content on the website was edited in a staging directory using a CMS (a fairly dated version of <a href="http://www.cmsmadesimple.org/">CMS Made Simple</a>), and then published using a <code class="language-plaintext highlighter-rouge">publish.php</code> script which copied the content from staging to production.</p>
<p>It turned out that the issue with the blank pages was with the publish script. Somehow, running the script, instead of copying pages from staging to production, had replaced the production pages with empty strings.</p>
<p>The copying was done with the following code:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>function getUrlContents($url, $credentials="") {
if ($credentials == "") {
return file_get_contents($url, false);
} else {
$context = stream_context_create(
array(
'http' => array(
'header' => "Authorization: Basic " . base64_encode($credentials)
)
)
);
return file_get_contents($url, false, $context);
}
}
</code></pre></div></div>
<p>My intial thought was that the issue was with <a href="http://php.net/manual/en/function.stream-context-create.php"><code class="language-plaintext highlighter-rouge">stream_context_create</code></a>, possibly with the <code class="language-plaintext highlighter-rouge">http</code> on line 7. But reading the documentation for that, and for <a href="http://php.net/manual/en/function.file-get-contents.php"><code class="language-plaintext highlighter-rouge">file_get_contents</code></a> and extensive googling for “stream context create HTTPS” got me nowhere. Eventually, I created a small test file to run this code locally. But it output the full page.</p>
<p>So then I deployed my test file to the server, and it failed there. I then tried <code class="language-plaintext highlighter-rouge">curl</code> on the server, and got this output:</p>
<p><code class="language-plaintext highlighter-rouge">curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure</code></p>
<p>A <a href="https://stackoverflow.com/questions/34578308/how-to-solve-curl-ssl-v3-alert-handshake-failure">StackOverflow answer</a> led me to question the version of OpenSSL:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>OpenSSL> version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
</code></pre></div></div>
<p>Essentially, the version of OpenSSL that <code class="language-plaintext highlighter-rouge">curl</code> was using doesn’t allow copying over HTTPS.</p>
<p>Unfortunately, with shared hosting, there’s nothing I can do about that. I can’t upgrade OpenSSL, and I certainly can’t rebundle <code class="language-plaintext highlighter-rouge">curl</code> and whatever <code class="language-plaintext highlighter-rouge">file_get_contents</code> is using.</p>
<p>The only solution to allow publishing using the current method was to allow the site to be served as HTTPS and HTTP.</p>
<p>This meant I had to change the settings on Cloudflare to no longer be “always use HTTPS”, which also means that I couldn’t use <a href="https://www.owasp.org/index.php/HTTP_Strict_Transport_Security_Cheat_Sheet">HSTS</a>. This was very annoying as it meant I then had to rely on the user to try to use HTTPS in order to protect themselves. On the plus side, it did allow me to publish the content again rather than an empty string.</p>
<h2 id="fixing-the-css">Fixing the CSS</h2>
<p>The issue with the CSS was that in the page source there was a tag <code class="language-plaintext highlighter-rouge"><base href="http://spaconference.org/" /></code>. I could not see where this was being set. After much investigation and a long time, it turned out that this was a metadata tag set by the CMS.</p>
<p>It turned out that to change this setting, I had to change some code in the template from <code class="language-plaintext highlighter-rouge">{metadata}</code> to <code class="language-plaintext highlighter-rouge">{metadata showbase="false"}</code>.</p>
<p><img src="/img/meta_tag.png" alt="Documentation about the metadata tag" /></p>
<p>It was not possible to find this by grepping; it involved understanding how the CMS worked. Not for the first time, I thought that I must remove the CMS.</p>
<p>Fixing this meant that the site was available over HTTPS, with CSS, but only if the user went to <code class="language-plaintext highlighter-rouge">https://</code> rather than <code class="language-plaintext highlighter-rouge">http://</code>. Not ideal, but a step in the right direction.</p>
<h2 id="a-series-of-embarrassing-roll-backs">A series of embarrassing roll-backs</h2>
<p>However, when it came to launch the 2018 site, I had to roll back even further.</p>
<p>The first roll-back I mentioned above was almost immediate, switching off HTTPS by default in July 2017, in theory until such time that I could fix the publishing/remove the CMS.</p>
<p>Then, in September 2017, when I was deploying the site for 2018, the whole thing went wrong. Instead of the site, I got a Cloudflare warning page.</p>
<p><img src="/img/cloudflare_warning.png" alt="Cloudflare warning" /></p>
<p>I needed to finish getting the 2018 site out, and didn’t have time to investigate and fix (often the way on side projects when the next priority should be going to bed). In order to move foward, I had to switch off HTTP entirely, which would have been an embarrassing full climbdown if anyone paid attention to the site outside of the conference season.</p>
<p>I couldn’t remove the CMS immediately, as I was in the middle of an <a href="/jfdi/removing-mediawiki-site-changes.html">epic of removing MediaWiki from the site</a>. While I was writing up that epic, I realised that the session and user pages on the 2017 site didn’t have CSS because I’d optimistically updated the links to point to HTTPS. So I then had to <a href="https://github.com/spaconference/previous-spa-sites/commit/33c5f1a6">find and replace all those links</a>.</p>
<p>However, the new site was being published with an HTTPS base tag: <code class="language-plaintext highlighter-rouge"><base href="https://spaconference.org" /></code> which meant that the CSS could not be retrieved as it was no longer available over HTTPS. I knew that I’d set that somewhere but I couldn’t remember where, and I spent a long time trying to figure that out. I eventually found the answer… above, in a draft of this blog post. (A huge benefit of <a href="/jfdi/how-to-blog.html">having a blog when you don’t have an amazing memory</a>!)</p>
<p>Then I realised even my new 404 and 500 error pages, added as part of the <a href="/jfdi/removing-mediawiki-cool-uris.html">MediaWiki removal</a> had a link to the HTTPS version of the homepage. Even that had to be removed.</p>
<p>It was a series of ever more embarrassing roll-backs until, in October 2017, I was exactly where I had been when the session was proposed.</p>
<h2 id="a-side-note">A side note</h2>
<p>At some point during this process, while looking at the certificate, I saw this.</p>
<p><img src="/img/depressing.png" alt="Shows I have visited site 1,920 times" /></p>
<p>That’s depressing.</p>
<h2 id="fixing-the-staging-issue">Fixing the staging issue</h2>
<p>In October, once I’d finished removing MediaWiki, I returned to this problem.</p>
<p>The current status was that I could only deploy from staging to live via HTTP because in order to do this, I had to copy the contents of the staging pages to production which didn’t work with the available version of OpenSSL. It seemed to me there were three potential ways to address this:</p>
<ol>
<li>Move staging from <code class="language-plaintext highlighter-rouge">http://spaconference.org/staging/</code> to <code class="language-plaintext highlighter-rouge">http://staging.spaconference.org</code>. This would mean I could put <code class="language-plaintext highlighter-rouge">spaconference.org</code> and <code class="language-plaintext highlighter-rouge">www.spaconference.org</code> on HTTPS while leaving <code class="language-plaintext highlighter-rouge">staging</code> HTTP to allow the data to be copied.</li>
<li>Find another way to copy from staging to production that doesn’t depend on OpenSSL, i.e. defer the problem for now with an interim fix until I’m ready to tackle removing the CMS.</li>
<li>Bring forward my plan to sack the CMS from ‘someday’ to ‘now’.</li>
</ol>
<p>I dismissed the first idea pretty quickly. It would be a lot of work for what would ultimately be a temporary solution. If I’m going to do a lot of work at this stage, it should be to remove HTTP entirely.</p>
<p>The third idea was clearly the best long term idea – removing the CMS was something I wanted to do anyway, but it would be a lot of work, and I wanted not to go down the rabbithole and just get the HTTPS back. So I looked closely at option 2.</p>
<h2 id="an-interesting-diversion">An interesting diversion</h2>
<p>My initial supposition was that the files would be generated on the server so I could just copy them over, but they’re not. CMS Made Simple stores the data in separate blobs in the database, and composing them is difficult. However, CMS Made Simple is open source, and because we are on shared hosting, the code was vendored, so I could dig into it and work out how the files were composed.</p>
<p>This was a very interesting diversion and involved a lot of diving into <a href="https://secure.php.net/manual/en/book.outcontrol.php">output buffering in PHP</a>. I spent quite some time down this rabbithole, dealing with issues like wanting my script to call an output buffer but it not working, and this being because there was an unexpected <code class="language-plaintext highlighter-rouge">ob_get_clean()</code> in the CMS Made Simple PHP; the buffers are not nested so that call clears all buffers for the rest of that process.</p>
<p>I finally managed to generate the <a href="https://spaconference.org/spa2017/lead-a-session.html">Lead a session</a> page using the following code.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code><?php
require_once('private_scripts/incl_file_helpers.php');
$alias = 'lead-a-session';
function buffer_callback($buffer_contents) {
$buffer_contents = preg_replace("/index\\.php\\?page=([^\\\"]*)\\\"/", "\\.html\"", $buffer_contents);
fileWrite("output.php",$buffer_contents);
return "$buffer_contents";
}
$_SERVER['REQUEST_URI'] = "/staging/index.php/" . $alias;
ob_start('buffer_callback');
include('index.php');
ob_end_flush();
?>
</code></pre></div></div>
<p>I could then call this by:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>`if (!isset($_SERVER['REQUEST_URI']) && isset($_SERVER['QUERY_STRING']))`
`QUERY_STRING='page=lead-a-session' php -q -d error_reporting=0 index.php > test.html`
</code></pre></div></div>
<p>So essentially the script recreates the page in the same way the CMS does for display, and then copies that page to production.</p>
<p>And this worked! I could use this as the publish script.</p>
<p>Except… it was missing the menu, which is constructed differently.</p>
<p>And a number of other things:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Remaining TODO for generating CMS pages
Needs nav and other sorting
Needs programme
Needs not to output the output of the script!
Needs to suppress header warning
Needs switching to HTTPS
Clear the extra buffer handlers
</code></pre></div></div>
<p>At this point, I gave up. This was, again, turning into a rabbithole for an uncertain and interim outcome. It was time to face facts and remove the CMS.</p>
<p>This was a huge piece of work that took several months (as this is something I maintain in my not hugely copious spare time), and I’ve written that up <a href="/jfdi/removing-cms-from-spa.html">in a separate post</a>.</p>
<h2 id="finally-having-removed-the-cms">Finally, having removed the CMS…</h2>
<p>I switched HTTPS back on. I had only paused it in Cloudflare, so this wasn’t a lot of setting back up again.</p>
<p>This time round, there was no issue with the CSS as it was relative, and the <a href="https://github.com/spaconference/spa-website/commit/da71c2a">base tag had been removed</a>. There was no mixed content for the same reason, all neater.</p>
<p>I turned ‘Always use HTTPS’ back on. This redirects all HTTP requests to HTTPS, which is an extra hop, so ideally I want to update all links in the <a href="https://www.spaconference.org/spa2018/previous-conferences.html">previous sites</a> but this is not a high priority.</p>
<p>Some of the previous sites don’t look great, but there is a Cloudflare setting ‘Automatic HTTPS Rewrites’ which helps fix mixed content by changing HTTP to HTTPS for all resources that can be served with HTTPS, which fixes some of them. I’ll update the others when I have time.</p>
<p>I changed back my new error pages and all the URLs in the code, and then switched HSTS back on.</p>
<p>Moving the site to HTTPS caused the build of the <a href="/jfdi/removing-cms-from-spa.html">Jekyll site that replaced the CMS</a> to fail in a way I hadn’t anticipated, but the amazing advantage of the static pages now being open meant that <a href="https://github.com/spaconference/spa-website/pull/25">someone else identified and fixed this problem</a> before I was aware of it.</p>
<p>The main issue outstanding is that the local version of the scripts doesn’t work over HTTPS. This is annoying, but currently only for me as I’m the only person working on it, so I’ll fix it at some point. I also set up the SPA account with Cloudflare under my account rather than a separate one, which will make it harder for me to hand over when I do, but that’s also a story for another day.</p>
<p>The great success is I finally managed to switch HTTPS back on in time for this year’s conference.</p>
<h2 id="im-not-sure-it-was-that-easy">I’m not sure it was that easy</h2>
<p>So, despite Yo and Nelis’s claims, it was not in fact that easy. To be fair, they didn’t know the full story of the 20 year old PHP site. And it’s a great relief to have done it.</p>
<p>I look forward to the next epic task they set me.</p>
Replacing the SPA CMS2018-07-20T00:00:00+00:00https://www.annashipman.co.uk/jfdi/removing-cms-from-spa.html<p>As part of my <a href="/jfdi/removing-mediawiki-cool-uris.html">ongoing mission</a> to update the <a href="https://spaconference.org">SPA conference website</a>, I had long wanted to replace the CMS. When I realised it was blocking the move to HTTPS, I knew the time was now. This post is about how I did that and what I learned.</p>
<h2 id="the-conference-website-used-a-cms-to-generate-the-static-pages">The conference website used a CMS to generate the static pages</h2>
<p>The functionality of the conference cycle, for example programme submission, reviews etc, is done through PHP scripts, but the generation of the static pages used <a href="http://www.cmsmadesimple.org/">CMSMadeSimple</a>.</p>
<p>The CMS didn’t edit the site directly, it edited a staging site, which was in fact a subdirectory: <code class="language-plaintext highlighter-rouge">spaconference.org/staging/</code>. When the chairs were happy with the content in staging, they published it to the live site (another subdirectory, e.g. <code class="language-plaintext highlighter-rouge">spaconference.org/spa2018</code>) using a script called <code class="language-plaintext highlighter-rouge">publish.php</code>.</p>
<h2 id="it-wasnt-very-secure">It wasn’t very secure</h2>
<p>The <code class="language-plaintext highlighter-rouge">staging</code> subdomain and and the <code class="language-plaintext highlighter-rouge">publish</code> script were behind <a href="https://tools.ietf.org/html/rfc7617">Basic Auth</a>, which when coupled with the fact that the site was still HTTP (more on that another day) meant it <a href="https://security.stackexchange.com/a/990">wasn’t very secure</a>. When new chairs joined the committee, I had to create their passwords for Basic Auth as I was the only one with access to <code class="language-plaintext highlighter-rouge">.htpasswd</code>, and email the passwords to them; very insecure. In addition, the <code class="language-plaintext highlighter-rouge">publish</code> script had the previous web admin’s credentials in it to allow it to work, which was a bit of a blocker to my plan of eventually open sourcing this code.</p>
<p>However, this is not a very high-profile website so the risk and impact of someone unauthorised gaining access is quite low. The CMS, while a bit dated, was relatively intuitive and easy for new chairs to use, so replacing it was not a high priority.</p>
<h2 id="it-was-blocking-the-move-to-https">It was blocking the move to HTTPS</h2>
<p>It became a priority to remove it when I discovered it was blocking moving the site to HTTPS. The issue wasn’t the CMS itself, but the <code class="language-plaintext highlighter-rouge">publish</code> script, which literally copied the page from the <code class="language-plaintext highlighter-rouge">staging</code> subdirectory to the current year subdirectory using <code class="language-plaintext highlighter-rouge">curl</code>. The version of OpenSSL running on the shared hosting didn’t support copying the page contents over HTTPS.</p>
<p>Because it’s shared hosting, it’s not possible for me to upgrade OpenSSL, so after considering various other ways to copy the site from staging to live, it became clear that the right solution was to bring forward my plan to remove the CMS and publish the site a different way.</p>
<h2 id="most-of-the-pages-were-static-so-could-be-replaced-with-a-jekyll-project">Most of the pages were static so could be replaced with a Jekyll project</h2>
<p>It was quite easy to choose <a href="https://jekyllrb.com/">Jekyll</a>. I’m familiar with it from using it on this blog, and I know others are too. <a href="https://github.com/spaconference/spa-website">Hosting the project on GitHub</a> meant that I could offload the issues around authorisation to GitHub, dealing with the security issues I mentioned above.</p>
<p>Using GitHub also has the advantage that people can see when others are editing and what changes are being published, and comment if neccessary. With the CMS, there was no way to tell if another user was editing at the same time. The <code class="language-plaintext highlighter-rouge">publish</code> script published everything, and a couple of the conference chairs had mentioned to me that this was a bit stressful, because there was no way of knowing if you were about to publish something half-finished that another chair was working on. With GitHub, you can see what commits have been made and what pull requests are open.</p>
<p>I also removed the necessity to run a separate <code class="language-plaintext highlighter-rouge">publish</code> script by setting the project up so that it deploys automatically to the live site on merge to master. This turned out to be quite tricky on shared hosting, so I <a href="/jfdi/shared-hosting-travis-deploy.html">wrote up how to do it</a>.</p>
<h2 id="the-programme-generation-was-a-different-kettle-of-fish">The programme generation was a different kettle of fish</h2>
<p>While most of the site content is completely straightforward, the <a href="https://spaconference.org/spa2017/programme.html">programme</a> generation was managed in a completely different way, not using the CMS. The programme was created via the PHP scripts, using a plug-in called <a href="http://trac.xinha.org/">Xinha</a> as a WYSIWYG editor. Because the site is on shared hosting, this was vendored into our code, and was an old version.</p>
<p><img src="/img/editing_programme_xinha.png" alt="Image showing how the edit programme with Xinha looked" /></p>
<p>The accepted sessions were available to be added via a drop-down menu. When chairs published the programme, the accepted session links were expanded to include all the details about the session and the session leaders, with links to the supplementary pages.</p>
<p>The session and session leader pages themselves had to be generated separately, using a <code class="language-plaintext highlighter-rouge">publishallsessions.php</code> script.</p>
<p>The process was not intuitive, but in theory it worked quite nicely because you could just add the sessions you wanted via a drop-down, and publishing it took care of generating all the rest of what was needed.</p>
<p>However, when I looked into it, it turned out that the last programme chairs who actually used it were <a href="https://spaconference.org/spa2014/programme.html">me and Andrew in 2014</a>. Subsequent chairs had generated or hand-written the HTML in a different way, including manually adding all the session details.</p>
<p>I wanted to create a process that had the good features of the old process (like automatically creating the session and session leader pages) but was easy enough to use that subsequent chairs would actually use it.</p>
<h2 id="the-programme-had-to-look-like-the-rest-of-the-site">The programme had to look like the rest of the site</h2>
<p>The way it had worked previously was that the “publish” button saved the HTML for the programme (so not the whole page, i.e. not the header and footer) to the file system on the server. If you look closely at the diagram above you can see the ‘Publish’ button says “to staging only”.</p>
<p class="half-width"><img src="/img/to_staging_only.png" alt="Close up of button that says 'Publish, to staging only'" /></p>
<p>The CMS then picked up the HTML from the file system, added the same header and footer as the rest of the site and published it along with everything else when the <code class="language-plaintext highlighter-rouge">publish</code> script was run. (This did mean that the preview and published programme HTML was actually available on the file system if you knew where to look, not even behind Basic Auth.)</p>
<p>Before launching the new Jekyll site, I did a proof of concept to make sure it would be possible to publish directly, rather than via the CMS.</p>
<p>I also looked around to see how other conferences generated their programmes, but I couldn’t find a good tool.</p>
<h2 id="working-out-how-to-generate-the-programme-took-ages">Working out how to generate the programme took AGES</h2>
<p>It took a very long time to turn the proof of concept into a working process for generating the programme because there was a lot to unpick. The main challenge was making it look like the rest of the site.</p>
<p>I didn’t want to add it to the Jekyll project, because that is public and I think it’s important that the programme generation is done privately. You wouldn’t want session proposers to find out their session had been rejected by being able to observe you editing the draft programme. You might also not want it to be public who had pulled out of the conference or other changes you might make in the run up to publishing it.</p>
<p>I considered a number of options for how to do it, and concluded that adding it directly to the correct place on the filesystem on publish was the least worst option.</p>
<h2 id="how-programme-generation-works-now">How programme generation works now</h2>
<p>Instead of generating HTML and saving that to the file system, my changes saved the programme into the database. I created a very basic form for editing.</p>
<p><img src="/img/new_editing_programme.png" alt="Basic form for editing programme" /></p>
<p>I had big plans for making it much prettier, but meanwhile the conference cycle marches on and I had to get something out before the programme meeting in March of this year, so MVP it is.</p>
<p>To keep it in synch with the Jekyll site, I added <a href="https://github.com/spaconference/spa-website/tree/master/programme_includes">programme includes</a>. These are included in the programme page so that when the rest of the site is generated they will change too – so, for example, if another item is added to the <a href="https://github.com/spaconference/spa-website/blob/master/_includes/menu.html">menu</a> this will show up the same on the programme page.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code><?php include('{$GLOBALS['pathToCurrentYear']}/programme_includes/header.html'); ?>
</code></pre></div></div>
<p>This means that when I’m retiring the site to go in the list of <a href="https://github.com/spaconference/previous-spa-sites/blob/master/README.md">previous conferences</a>, I’ll need to make sure to replace that with the actual HMTL.</p>
<p>I also added the ability to preview the page exactly as it would look on the site.
<img src="/img/preview_programme.png" alt="previewing programme looks as it would on site" /></p>
<p>And the publish button now actually publishes to the live site, by saving it in the correct place on the filesystem.</p>
<p>I’m not very happy with the separation of the generation of the static pages from the generation of the programme, especially as there is a chance they could be briefly out of synch (e.g. if you add a new menu item and republish the site but do not republish the programme). In practice, this will rarely happen as once the programme is published it tends to be frequently republished as things change, e.g. session leaders update details of their sessions. However, I see this as an area to be improved in future.</p>
<p>The preview and publish code both reused the code from the previous editing of the programme to generate the full info for the session, including a link to the session page and the user pages.</p>
<p>In addition, I made it so that publishing the programme also publishes those session and user pages rather than requiring the extra step of the <code class="language-plaintext highlighter-rouge">publishallsessions.php</code> script..</p>
<p>(This meant, before my changes, if a user updated their bio and asked you to publish it you would have to run <code class="language-plaintext highlighter-rouge">publishallsessions.php</code> in the browser, then navigate to the edit programme page in the site and hit the <code class="language-plaintext highlighter-rouge">publish (to staging only)</code> button, and then run <code class="language-plaintext highlighter-rouge">publish.php</code> in the browser. Now, you just navigate to the programme and hit the <code class="language-plaintext highlighter-rouge">publish</code> button.)</p>
<h2 id="tidying-up-after-the-programme">Tidying up after the programme</h2>
<p>Finally, I deleted all the Xinha code. This was INCREDIBLY SATISFYING.</p>
<p><img src="/img/tweet_delete.png" alt="Screenshot of https://twitter.com/annashipman/status/956164933280034816" /></p>
<h2 id="tidying-up-after-the-cms">Tidying up after the CMS</h2>
<p>But that was NOTHING compared to the joy of getting rid of the CMS. Not just the code – that was great – but also tidying the database.</p>
<p>CMSMadeSimple stores everything in the database, so once I’d removed it I could remove 54 tables. Added to the <a href="/jfdi/removing-mediawiki-cool-uris.html">removal of the MediaWiki tables</a>, in a few months I had gone from a database with 113 tables to one with 10. This is so much better.</p>
<p>Also, because the <code class="language-plaintext highlighter-rouge">publish.php</code> script copied everything from staging to live, it meant that there were several unused files that got copied over year after year. For example (check out the URL) <a href="https://spaconference.org/spa2010/uploads/images/spa2009/headersnaps/1.jpg">https://spaconference.org/spa2010/uploads/images/spa2009/headersnaps/1.jpg</a>. And here it is again in 2017: <a href="https://spaconference.org/spa2017/uploads/images/spa2009/headersnaps/1.jpg">https://spaconference.org/spa2017/uploads/images/spa2009/headersnaps/1.jpg</a>… and every year in between. It’s nice to tidy those things up.</p>
<h2 id="making-it-easier-to-develop-locally">Making it easier to develop locally</h2>
<p>Removing the CMS also made it much easier to set up a local development environment. When I took over running the website in 2014, I did what I always do: set up a Vagrant box, managed by Puppet, so I have to do as little manual set up as possible.</p>
<p>Part of our setup of the CMS involved some modifications to the staging site, so I set up Puppet to copy over those modifications. These were things like the <code class="language-plaintext highlighter-rouge">.htaccess BasicAuth</code>. Removing the CMS meant I could remove a whole Puppet module, and also some manual instructions for setting up the site locally.</p>
<p>It also meant that developing locally didn’t require setting up a local version of the CMS.</p>
<h2 id="what-did-the-new-chairs-think">What did the new chairs think?</h2>
<p>I rushed to get this ready so that this year’s programme chairs could set up the programme for this year. They did this, using the site, and didn’t ask me any questions about it so I assumed it was straightforward enough to use.</p>
<p>At the conference, I managed to grab them to get their feedback, and they hadn’t felt quite as positive about it as I’d hoped. Initial response was “completely incomprehensible interface”.</p>
<p>However, they managed to use it rather than handwriting HTML, unlike any other chair since 2014, so I still call it a win.</p>
<p>We came up with some clear ideas of how to improve the interface to make it more useful and easier to use. However, one thing worth mentioning is that a lot of the programme chairs’ concerns were that it didn’t allow them to add all the information involved in generating the programme, like the speakers’ availability, their AV requirements, etc. That is by design; the aim of this form is to give chairs an easy way to generate the programme in the style of the rest of the site. The complexity of putting the programme together needs to be done elsewhere because we can’t assume chairs will do it in the same way. Some might use Trello, some spreadsheets, some index cards, etc.</p>
<p>However, I’m definitely going to make some improvements to make it more user-friendly to do the job it’s meant to do. Now it is in my control, rather than using an external tool, I can continue to iterate it to make it better.</p>
<h2 id="was-it-worth-it">Was it worth it?</h2>
<p>If this had been part of my job, it would have been very hard to justify doing this work, especially as it took so long. The right solution would probably have been to buy a tool that doesn’t reflect our process, and change our process.</p>
<p>But this is a side project for me, and I enjoy tinkering and improving it, and in doing so it allows us to support our <a href="https://www.spaconference.org/spa2018/submission-stages.html">community-led and anonymous submission process</a>.</p>
<p>And it is really satisfying to be able to do things that people like, and get that feedback. One of the programme chairs said the CMS made her feel stressed because she didn’t know what needed changing, but with the new Jekyll site, she could see what everything was and it felt “calming”. That is high praise indeed. When it’s all code, you can <code class="language-plaintext highlighter-rouge">grep</code>, but with the CMS you might not be able to find what you are looking for.</p>
<p>And another huge advantage of making the static pages open and accessible to others is that when something on the site broke, in this case the build, it wasn’t just an email to me to do something; someone else could <a href="https://github.com/spaconference/spa-website/pull/25">fix it</a>.</p>
<p>Roll on open sourcing the rest of it!</p>
Understanding my strengths using a Johari window2018-06-02T00:00:00+00:00https://www.annashipman.co.uk/jfdi/johari-window.html<p>When leaving my last job I decided to do an exercise which compares what other people think my strengths are with what I think my strengths are. This is a write-up.</p>
<h2 id="others-may-know-you-better-than-you-know-yourself">Others may know you better than you know yourself</h2>
<p>Various studies over the years have found that your colleagues are actually better at identifying some of your strengths than you are yourself. Here’s an <a href="https://www.theguardian.com/money/2005/dec/03/workandcareers.careers">interesting article in the Guardian from 13 years ago</a> and a more recent <a href="https://www.theatlantic.com/health/archive/2018/03/you-dont-know-yourself-as-well-as-you-think-you-do/554612/">article in the Atlantic</a>.</p>
<p>The Guardian article suggests you try an exercise called a <a href="https://en.wikipedia.org/wiki/Johari_window">Johari window</a>, and when I was leaving GDS seemed like a good time to try it out.</p>
<p>To do this, you ask some co-workers to choose 5 or 6 adjectives that they think best describe you, from a list of 56. You do the exercise yourself separately, and then you compare your answers with theirs.</p>
<p>Adjectives you select and they also select are strengths that you have self-awareness of. Things they select and you don’t, are traits that are obvious to others and not to you. Adjectives you select and they don’t are things that aren’t as obvious to others as they are to you – maybe because you hide them (e.g. you might try and hide anxiety) or perhaps because you don’t know yourself as well as you think.</p>
<h2 id="i-set-the-exercise-up-using-a-google-form">I set the exercise up using a Google form</h2>
<p>I listed the adjectives in the order they are presented in the Guardian article (rather than Wikipedia which has them in alphabetical order). There is one adjective on Wikipedia (empathy) that is not in the Guardian so I added that. The form was anonymous, so I wouldn’t know who had said what.</p>
<p>I then emailed 11 people I’d worked with closely over the past few years and asked them to fill it in, with a link to the Guardian article for more explanation.</p>
<p><img src="/img/johari_google_form.png" alt="Screenshot of the Google form" /></p>
<p>For my own results, I went through the list deleting until I had the set. It was pretty hard to get it from around 12 to 6 and I had to do a lot of hard thinking (“Am I more this than that?”).</p>
<p>I also found that some of my strengths didn’t seem to be represented. For example, one of my strong points is that I get things done: I’d characterise this as determination or single-mindedness. What adjective in the list would represent that?</p>
<h2 id="i-put-the-results-into-a-johari-window">I put the results into a Johari window</h2>
<p>Of the 11 people I asked, 10 filled it in. Here is a photo of the results. The number in brackets refers to the number of other people (not including me) who selected it.</p>
<p>If you can’t read my handwriting or see the picture, I’ve listed the full results at the end.</p>
<p><img src="/img/johari_results.png" alt="Image of the results. Full details and results listed in appendix" /></p>
<h2 id="the-results-were-positive-and-interesting">The results were positive and interesting</h2>
<p>This seems to me to be a good result; of the 6 strengths that I identified, 5 of them were agreed with by others, showing that I have a good amount of self-awareness. Additionally, the ones that I value most highly in myself (intelligence, trustworthiness) were selected by a high number of the others, which is pleasing. The one that I selected and no-one else did was “caring” which I dithered over whether I meant “caring” or “kind” – and 2 people selected “kind”.</p>
<p>There were some interesting results. Three people picked “calm”, which isn’t something I’d use to describe myself (it didn’t make the top 12). But does it really matter? In a work context, if people think I am calm then that’s a good thing, whether or not I consider myself to be calm on the inside.</p>
<p>Some of them also seem very similar. What’s the difference between “independent” (which 4 picked and I did not) and “self-assertive” (which I picked and 2 others also picked)? More interestingly, what’s the difference between “intelligent” (which I and 5 others picked) and “clever” (which no-one picked)? Is the difference there just a factor of “intelligent” being earlier in the list?</p>
<p>I’m not sure of the value of the fourth quadrant, “Not known to self or others” - it might just not be known because it isn’t true. This is referred to as the untapped area, but some of them you wouldn’t necessarily want to tap, like “silly”, or “tense”. And some of them contradict others, e.g. “introvert”/”extravert”, so you’d hope for at least one of those in the fourth quadrant. I think this is where the model doesn’t quite work, but I haven’t read the original paper.</p>
<h2 id="if-i-did-it-again-i-might-do-a-few-things-differently">If I did it again I might do a few things differently</h2>
<p>The Google form was not very user-friendly. If I was doing it again, I think I would print out pieces of paper and ask people to circle the relevant ones, as that might be a much easier way for them to parse all the information and pick one. It is a lot to ask of people, so anything that could make it easier would be helpful.</p>
<p>I’d also be tempted to remove some of the options, both to make it easier for the participants, and also because some of them are odd. For example “loving” is not something I would expect colleagues to comment on. “Religious” is odd as well, is that relevant to your personality? Could we remove some of the near duplicates, like “intelligent”/”clever”?</p>
<p>Another possibility was suggested to me by my excellent art therapist and trainer friend, Dr Sarah Haywood: you could just ask people to supply their own adjectives and then put them into a word cloud or similar. That might also have the advantage of drawing out some more negative words, which is also really useful to know.</p>
<h2 id="im-glad-i-finally-did-it">I’m glad I finally did it</h2>
<p>I first saw the article in the newspaper when it came out, over a decade ago, and at the time I did not feel confident enough to ask colleagues to do that for me. That change in itself is interesting – I may be more self-aware than I was then, but I’m also much more comfortable with asking people to do things like this for me.</p>
<p>In all, it was an interesting and worthwhile exercise, and it was nice to see that my self-assessment is reasonably close to what others think of me. If you do try it for yourself, do let me know how you found it!</p>
<h2 id="appendix-full-results">Appendix: Full results</h2>
<p>The number in brackets refers to the number of other people (not me) who selected it.</p>
<h3 id="known-to-self-and-others">Known to self and others</h3>
<ul>
<li>Intelligent (5)</li>
<li>Trustworthy (4)</li>
<li>Organised (3)</li>
<li>Self-assertive (2)</li>
<li>Wise (2)</li>
</ul>
<h3 id="known-to-others-and-not-known-to-self">Known to others and not known to self</h3>
<ul>
<li>Knowledgeable (5)</li>
<li>Independent (4)</li>
<li>Able (3)</li>
<li>Logical (3)</li>
<li>Calm (3)</li>
<li>Dependable (2)</li>
<li>Sensible (2)</li>
<li>Kind (2)</li>
<li>Confident (2)</li>
<li>Friendly (2)</li>
<li>Introverted (2)</li>
<li>Responsive (2)</li>
<li>Helpful (2)</li>
<li>Energetic (2)</li>
<li>Patient (1)</li>
<li>Quiet (1)</li>
<li>Empathetic (1)</li>
<li>Cheerful (1)</li>
<li>Observant (1)</li>
<li>Warm (1)</li>
</ul>
<h3 id="known-to-self-and-not-others">Known to self and not others</h3>
<ul>
<li>Caring</li>
</ul>
<h3 id="not-chosen-by-anyone">Not chosen by anyone</h3>
<ul>
<li>Accepting</li>
<li>Dignified</li>
<li>Powerful</li>
<li>Sentimental</li>
<li>Adaptable</li>
<li>Proud</li>
<li>Shy</li>
<li>Bold</li>
<li>Extraverted</li>
<li>Silly</li>
<li>Brave</li>
<li>Spontaneous</li>
<li>Giving</li>
<li>Loving</li>
<li>Relaxed</li>
<li>Sympathetic</li>
<li>Happy</li>
<li>Mature</li>
<li>Religious</li>
<li>Tense</li>
<li>Modest</li>
<li>Clever</li>
<li>Idealistic</li>
<li>Nervous</li>
<li>Searching</li>
<li>Complex</li>
<li>Ingenious</li>
</ul>
Cleaning up SPA for GDPR2018-05-20T00:00:00+00:00https://www.annashipman.co.uk/jfdi/gdpr.html<p>I think GDPR is great. I am really pleased that companies are now being more thoughtful about how they handle my data. For SPA, a conference that has been running for over 20 years, this coincided with a project I had already started to clean up the user data, and here’s what I did.</p>
<h2 id="we-had-a-lot-of-dormant-users">We had a lot of dormant users</h2>
<p>When I first decided to clean up the user data last summer, there were 1176 registered users, of whom 906 hadn’t logged in in the past 5 years. Some of them hadn’t logged in since the account was migrated from a previous database in 2004.</p>
<p>The data we store about users is mainly the information they enter in their profile: name, email address, and maybe bio, company name, Twitter handle etc. We used to also store address, phone number and fax (!) details if the user entered them, but I removed these fields and all the data <a href="https://www.annashipman.co.uk/jfdi/removing-mediawiki-site-changes.html">as part of retiring the wiki</a>.</p>
<p>We also store proposals users have made, and any feedback or reviews of other proposals, but only for the duration of one conference cycle (i.e. they are deleted every year).</p>
<h2 id="first-step-delete-a-large-number-of-inactive-users">First step: delete a large number of inactive users</h2>
<p>As Mags Allen points out in this <a href="https://medium.com/ft-product-technology/a-developers-guide-to-gdpr-that-won-t-make-you-sweat-4f1f7f1d9c8b">excellent developers’ guide to GDPR</a>, one of the key points of GDPR is only to store the data you need. So the first step was easy; I deleted everyone who hadn’t logged in since 2010. I left a long gap as, because of the community nature of SPA, people might not submit for a few years and then come back into the fold.</p>
<p>Although it wasn’t completely straightforward; the dates were stored as strings and the format had been changed at some point from <code class="language-plaintext highlighter-rouge">d-M-Y</code> to <code class="language-plaintext highlighter-rouge">j F Y</code> (i.e. from <code class="language-plaintext highlighter-rouge">20-Nov-2004</code> to <code class="language-plaintext highlighter-rouge">20 November 2004</code>) which made identifying the earlier records to delete somewhat complicated. (Something like the below helped).</p>
<p><code class="language-plaintext highlighter-rouge">SELECT * FROM users where str_to_date( reg_date, '%d-%M-%Y' ) > '20107-01-01'</code></p>
<p>I changed the format back, so this will even itself out over time as users either log in or are deleted for being inactive.</p>
<p>This left me with 584 users.</p>
<h2 id="working-out-what-we-need-consent-for">Working out what we need consent for</h2>
<p>We generally only email our users a few times a year, announcing the CFP, asking them if they’d like to get involved in giving feedback or reviewing sessions, and then to tell them about the conference and suggest they come.</p>
<p>The first two are arguably “legitimate interest” because we have a “relevant and appropriate relationship” to the people on the list, but the last one is definitely marketing. In any case, I think it’s best to be explicit.</p>
<p>So in November last year, I added a checkbox to ask for consent to receiving emails. I highlighted the box so that users who hadn’t seen it would have their attention drawn to it.</p>
<p><img src="/img/consent_checkbox.png" alt="Checkbox with label text "Can we email you about the conference? We send occasional emails, for example announcing our call for papers each year. Check this box if that's OK"" /></p>
<p>I also made it easier for new users to register. Previously when signing up for an account, you were asked to fill in a whole load of stuff including a bio. In fact none of this info was needed unless you were a session leader, and it was offputting and a hassle to be asked for it all up front.</p>
<p>I changed this so on first registering you are now only asked for name, email and consent to contact. If you come to edit your profile later you are given the option then to enter the rest of the info. In most cases, you would only edit your profile if your session is accepted to the conference, at which point details such as bio become relevant.</p>
<h2 id="you-could-see-all-the-registered-users">You could see all the registered users</h2>
<p>Because SPA is mostly interactive workshops, many sessions have more than one session leader. One person submits, and then adds the people they will be co-leading with.</p>
<p>If they were new to SPA, you added their email address and an account was created for them. But if they weren’t new to SPA, you could select their name from a drop-down list of all registered users.</p>
<p><img src="/img/choose_from_list.png" alt="all users in drop down" /></p>
<p>Like many aspects of the conference, this dates back from a couple of decades ago when it was mostly a group of people who already knew each other. But fast-forward to the current situation with over 1000 users and being able to enumerate all users is really not great security. It’s also not a great user experience.</p>
<p>I changed this so that you just add the co-leader’s name, and behind the scenes we figure out if they are an existing user or not.</p>
<h2 id="i-tested-the-current-list">I tested the current list</h2>
<p>Once I’d made those changes, in early December last year, I emailed all the remaining users to announce our call for papers, including an unsubscribe link and instructions on how to delete their account. This was the first time we’d emailed the list for a while.</p>
<p>It went to 584 people, had 95 hard bounces (including an entire company!) and 5 unsubscribes which helped clear up the list. (This, among other publicity, resulted in some excellent submissions for the conference this year and it’s a great <a href="https://www.spaconference.org/spa2018/programme.html">programme</a> – you should come!)</p>
<h2 id="four-months-later-i-reviewed-who-had-consented">Four months later, I reviewed who had consented</h2>
<p>By April this year, there were 553 users.</p>
<p>77 had said yes to being contacted, 15 had said no, and 461 had neither said yes or no (i.e. they hadn’t edited their profile).</p>
<p>Of the 77 who had said yes, 67 had registered after the addition of the consent checkbox – so would have seen it on the page they created the account.</p>
<p>Of the 15 who said no, 14 had also registed after the change.</p>
<p>It’s not possible for you to say neither yes nor no on first registration, because not checking it saves it as no. However, it is possible then to log into your account and not see it, because it is on an ‘edit my profile’ page which you may not visit if you’ve logged in for another reason.</p>
<p>In fact, of the 461 who still had a NULL value for the checkbox, 23 had logged in since the change but clearly hadn’t edited their profile. Not a huge number but worth making it more prominent, so I also flagged it up on the first page you see when you log in. Over the next couple of weeks, a few more people changed the setting for that.</p>
<p>It was also nice to note that the majority of new users (88%) had consented to email, which was encouraging about the messaging.</p>
<h2 id="last-chance-to-sign-up">Last chance to sign up</h2>
<p>Alongside this I’d been <a href="https://github.com/spaconference/spa-website/pull/47">updating our terms and conditions</a> with the help of the other committee members.</p>
<p>On 11th May I sent an email announcing the programme. I sent two versions: one to all those who’d checked yes to emails, and a different one to the remaining people who had selected neither yes nor no, which in addition to the details of the programme also said that it was the last email they’d receive from us unless they opted in.</p>
<p>This had a good result; within 24 hours the number of people consenting to contact had gone up to 92, and a week later it was up to 100.</p>
<h2 id="and-were-done-for-now">And we’re done for now</h2>
<p>There’s some extra stuff I’d like to do, like make it easier for users to delete their own accounts rather than having to ask one of us to do it, but I’m really happy to have cleaned up the user data so we are only storing the minimum information about people, and are only contacting those who are actually engaged with the conference.</p>
<p>And it’s a great conference, you should <a href="https://www.spaconference.org/">come</a>!</p>
How coding in the open can help you release faster2018-04-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/open-code-helps-you-release-faster.html<p>A few weeks ago I talked about how coding in the open can help you release faster at <a href="https://pipelineconf.info/">PipelineConf</a>. This was my last talk as Open Source Lead at the Government Digital Service. Here is the video, slides, links, and a bonus sketchnote!</p>
<h2 id="video-30-minutes">Video (30 minutes)</h2>
<div class="embedded">
<iframe src="https://www.youtube-nocookie.com/embed/M94Hg1mMGAg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
</div>
<h2 id="slides">Slides</h2>
<iframe src="//www.slideshare.net/slideshow/embed_code/key/m8SrwlRV2HmlhL" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe>
<h2 id="sketchnote">Sketchnote</h2>
<p>For a great summary, here is an excellent sketchnote of it by <a href="https://twitter.com/MrsSarahJones">Sarah Jones</a>.</p>
<p><img src="/img/Sarah_Jones_open_code_sketchnote.jpg" alt="Sarah Jones" /></p>
<p>I love to see sketchnotes of my talks, especially when they are as clear and nicely designed as this one. Thank you, Sarah!</p>
<h2 id="links">Links</h2>
<p>Finally, I mention a lot of resources in my talk. Most of them can be found in this <a href="/jfdi/open-code-resources.html">list of useful open code resources</a>. Those specific to this talk are below.</p>
<p><a href="https://spaconference.org">SPA Software in Practice conference</a> – you should totally come!</p>
<h3 id="examples-of-how-coding-in-the-open-helps-government">Examples of how coding in the open helps government</h3>
<p><a href="https://sfadigital.blog.gov.uk/2016/11/17/when-build-a-thing-really-works/">When build a thing really works</a> – blog post about using smart answers code.</p>
<p><a href="https://gdstechnology.blog.gov.uk/2018/01/05/how-making-our-deployment-code-open-improved-our-workflow">How making our deployment code open improved our workflow</a></p>
<p>Point 8 of the Government Service Standard: <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">Make all new source code open</a></p>
<h3 id="addressing-challenges-of-coding-in-the-open">Addressing challenges of coding in the open</h3>
<p><a href="https://github.com/alphagov/styleguides/blob/master/pull-requests.md">GDS pull request guidance</a></p>
<p>Blog post: <a href="https://gdstechnology.blog.gov.uk/2016/09/30/easing-the-process-of-pull-request-reviews/">Easing the process of pull request reviews</a></p>
<p>John Allspaw’s blog post about the importance of being able to recover quickly: <a href="https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/">MTTR is more important than MTBF (for most types of F)</a></p>
<h3 id="companies-that-open-code-even-when-its-their-secret-sauce">Companies that open code, even when it’s their secret sauce</h3>
<p><a href="https://feedbin.com/blog/2013/08/27/feedbin-is-open-source/">Feedbin is open source</a></p>
<p><a href="https://about.gitlab.com/2016/07/20/gitlab-is-open-core-github-is-closed-source/">GitLab uses LibGit2</a></p>
<h3 id="open-code-is-the-foundation-for-a-culture-of-openness">Open code is the foundation for a culture of openness</h3>
<p><a href="https://app.productplan.com/p/bUmH4fHC0hOivX-E2LYMu2hg9uEhkWp_">GOV.UK roadmap</a></p>
<p><a href="https://trello.com/b/7yWk0jhI/govuk-publishing-platform-tap-support-planning">Trello for the publishing platform team</a></p>
<p><a href="https://github.com/alphagov/govuk-rfcs">GOV.UK architectural decisions</a></p>
<p><a href="https://insidegovuk.blog.gov.uk/category/incident-reports/">GOV.UK incident reports</a></p>
Deploying to shared hosting with Travis2018-03-13T00:00:00+00:00https://www.annashipman.co.uk/jfdi/shared-hosting-travis-deploy.html<p>The <a href="https://www.spaconference.org/">SPA conference website</a> deploys to production automatically on merge to master. It is on shared hosting, and setting this up with <a href="https://travis-ci.org/">Travis</a> was a bit tricky, so I’m blogging how I did it in case it’s useful for others.</p>
<h2 id="the-documentation-on-deploying-to-shared-hosting-is-not-very-clear">The documentation on deploying to shared hosting is not very clear</h2>
<p>Configuring Travis to deploy automatically on merge to master is pretty straightforward if you’re using a <a href="https://docs.travis-ci.com/user/deployment/">supported provider</a>. However, if you’re using shared hosting, the documentation is not amazing.</p>
<p>I figured it out eventually by juggling between <a href="https://oncletom.io/2016/travis-ssh-deploy/">a blog post</a> (this one is useful for explanations of how Travis works but less so for step-by-step), the <a href="https://github.com/oncletom/oncletom.io/blob/master/.travis.yml">actual config that person uses</a>, <a href="https://andreas.heigl.org/2016/06/07/automated-deploy-from-travis-ci-via-ssh/">another blog post</a> (useful for some of the steps), the Travis docs on <a href="https://docs.travis-ci.com/user/encrypting-files/">encrypting files</a>, <a href="https://docs.travis-ci.com/user/deployment/custom/">custom deployment</a> and <a href="https://docs.travis-ci.com/user/customizing-the-build/#The-Build-Lifecycle">the build lifecycle</a>, and <a href="http://ajaykarwal.com/deploying-jekyll-using-travis-ci/">yet another blog post</a> (very useful for the actual config, but for a different set-up than I wanted).</p>
<p>That juggling made me think my addition to the genre was worth it.</p>
<h2 id="the-steps-i-took">The steps I took</h2>
<ol>
<li>Follow <a href="https://docs.travis-ci.com/user/getting-started/#To-get-started-with-Travis-CI">Travis’s get started instructions</a> to sign in to Travis and add the repository you want to deploy.</li>
<li>
<p>Create a <code class="language-plaintext highlighter-rouge">.travis.yml</code> file. The inital one can just have the language and how to build the site.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> language: ruby
rvm:
- 2.3.3
script: scripts/build.sh
branches:
only:
- master
</code></pre></div> </div>
<p>You can tell Travis how to build the site inline, or call out to a build script as I have done.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #!/bin/bash
set -e
bundle exec jekyll build
</code></pre></div> </div>
<p>The build script has to be executable (<code class="language-plaintext highlighter-rouge">chmod +x 600</code>).</p>
</li>
<li>Install travis locally (<code class="language-plaintext highlighter-rouge">gem install travis</code>), and, using the command line tool, <a href="https://docs.travis-ci.com/user/encryption-keys">encrypt</a> any variables that you will use for deploy. For example, if you use FTP, you will want to encrypt at least the password.
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> travis encrypt SOMEVAR="secretvalue" --add
</code></pre></div> </div>
<p>The <code class="language-plaintext highlighter-rouge">--add</code> flag in this command adds the <a href="https://github.com/spaconference/spa-website/blob/3ed4b7c9d647d5c3cc5d90f746c9549883699a1d/.travis.yml#L8-L11">required lines</a> to your Travis file.</p>
</li>
<li>
<p>Write a deploy script: a script that describes the steps you take to deploy to shared hosting. For example, if you use FTP, then write the command that you would use to FTP the built site to the server. Instead of the variables you have encrypted, use <code class="language-plaintext highlighter-rouge">SOMEVAR</code>.</p>
<p>I use Rsync to deploy, and have encrypted the username and deploy host. You can see <a href="https://github.com/spaconference/spa-website/blob/master/scripts/deploy.sh">the deploy script</a> on GitHub.</p>
<p>This also needs to be executable.</p>
</li>
<li>
<p>Create a dedicated SSH key (no passphrase) for deploying. This makes it easy to to identify and revoke if necessary.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ssh-keygen -t rsa -b 4096 -C 'build@travis-ci.org' -f ./deploy_rsa
</code></pre></div> </div>
</li>
<li>
<p>Log in to command line Travis (<code class="language-plaintext highlighter-rouge">travis login</code>) and get Travis to <a href="https://docs.travis-ci.com/user/encrypting-files/">encrypt the private key file</a>. It prints a helpful output reminding you to only commit the <code class="language-plaintext highlighter-rouge">.enc</code> version NOT the <code class="language-plaintext highlighter-rouge">deploy_rsa</code> itself.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> travis encrypt-file deploy_rsa --add
</code></pre></div> </div>
<p>Again, the <code class="language-plaintext highlighter-rouge">--add</code> flag automatically adds the <a href="https://github.com/spaconference/spa-website/blob/3ed4b7c9d647d5c3cc5d90f746c9549883699a1d/.travis.yml#L14-L16">required lines</a> to your Travis file.</p>
<p>Commit the changes to <code class="language-plaintext highlighter-rouge">.travis.yml</code> and the <code class="language-plaintext highlighter-rouge">deploy_rsa.enc</code>.</p>
</li>
<li>
<p>Copy the public key to the remote host.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ssh-copy-id -i deploy_rsa.pub <ssh-user>@<deploy-host>
</code></pre></div> </div>
</li>
<li>Delete the public key and the private key as they are no longer needed; you only need the encrypted key.
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> rm -f deploy_rsa deploy_rsa.pub
</code></pre></div> </div>
</li>
<li>
<p>In the <code class="language-plaintext highlighter-rouge">before_install</code>, change the <code class="language-plaintext highlighter-rouge">out</code> location for the decrypted private key to <code class="language-plaintext highlighter-rouge">/tmp/</code> so it doesn’t end up in any of the publically accessible directories, and make it executable.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> before_install:
- openssl [snip...] -out /tmp/deploy_rsa -d
- chmod 600 /tmp/deploy_rsa
</code></pre></div> </div>
</li>
<li>
<p>In the same section, start the ssh agent and add the key to it.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> - eval "$(ssh-agent -s)"
- ssh-add /tmp/deploy_rsa
</code></pre></div> </div>
</li>
<li>
<p>Get <code class="language-plaintext highlighter-rouge">.travis.yml</code> to call the deploy script.</p>
<p>This is an example of how the Travis docs could be clearer. You don’t want to deploy if the build script fails, so you might want to call this in an <code class="language-plaintext highlighter-rouge">after_success</code> step as part of a <a href="https://docs.travis-ci.com/user/deployment/script">script deployment</a>, but in fact, this will do whatever you ask after every successful build. But in our case, we only want to actually deploy on merges to master, not on <em>any</em> successful build. For example, if we’ve just opened a PR and that builds successfully, we don’t want that branch deployed to production.</p>
<p>So what we actually want is <a href="https://docs.travis-ci.com/user/deployment/custom/">custom deployment</a>.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code> deploy:
provider: script
script: scripts/deploy.sh
skip_cleanup: true
on:
branch: master
</code></pre></div> </div>
<p><code class="language-plaintext highlighter-rouge">skip_cleanup</code> is required because otherwise <a href="https://docs.travis-ci.com/user/deployment#Uploading-Files-and-skip_cleanup">Travis resets the working directory</a>, so you lose the artefects from the build (i.e. exactly what you want to deploy!)</p>
</li>
</ol>
<h2 id="what-my-config-looks-like">What my config looks like</h2>
<p>The full <code class="language-plaintext highlighter-rouge">travis.yml</code> we’ve now created looks like this:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>language: ruby
rvm:
- 2.3.3
script: scripts/build.sh
branches:
only:
- master
env:
global:
- secure: qvSoY270qAXOtmWdRio9vvhLEf5HHdyzMS39yS4yZw74[snip for length]
- secure: Hr7FV7lHFEblYfn7EYM/4qV3qV8zdHLebXzNyRvP8L/U[snip for length]
before_install:
- openssl aes-256-cbc -K $encrypted_ed2cb1b127e1_key -iv $encrypted_ed2cb1b127e1_iv
-in deploy_rsa.enc -out /tmp/deploy_rsa -d
- chmod 600 /tmp/deploy_rsa
- eval "$(ssh-agent -s)"
- ssh-add /tmp/deploy_rsa
deploy:
provider: script
script: scripts/deploy.sh
skip_cleanup: true
on:
branch: master
</code></pre></div></div>
<h2 id="further-information">Further information</h2>
<p>This (allegedly deprecated but still working) <a href="http://lint.travis-ci.org/"><code class="language-plaintext highlighter-rouge">travis.yml</code> linter</a> is useful.</p>
<p>Have a look at the <a href="https://github.com/spaconference/spa-website/blob/master/.travis.yml">SPA website Travis file</a> and <a href="https://travis-ci.org/spaconference/spa-website/builds/">the builds</a> if you’d like to know more about how it works for us.</p>
<h2 id="edit">Edit</h2>
<p>Useful commit for future reference is the <a href="https://github.com/spaconference/spa-website/commit/3ed4b7c9d6">creating/encrypting SSH key one</a>.</p>
Resources for coding in the open2018-03-02T00:00:00+00:00https://www.annashipman.co.uk/jfdi/open-code-resources.html<p>One of my main aims as <a href="/jfdi/a-year-in-the-life-os-lead.html">Open Source Lead</a> at the Government Digital Service was to make sure that there were good resources in place to help people code in the open. Many of these didn’t exist when I started the role so I created them. I’ve collected useful resources here.</p>
<h2 id="the-benefits-of-coding-in-the-open">The benefits of coding in the open</h2>
<p>I was often asked what the value of coding in the open is to the teams
themselves, those who are opening the code. There is a lot of value. I’ve
shared that in various formats:</p>
<ul>
<li>Blog post on GDS blog: <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">The benefits of coding in the open</a></li>
<li>30 min conference talk: <a href="https://www.turingfest.com/2017/engineering/anna-shipman?wvideo=0p810lpdqr">Coding in the open in government</a></li>
<li>2 min video: <a href="https://www.youtube.com/watch?time_continue=1&v=aqFFCvjXr1s">Why we code in the open</a></li>
</ul>
<h2 id="how-to-make-your-code-open">How to make your code open</h2>
<ul>
<li>Official guidance on the Government Service Design Manual: <a href="https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable">Making source code open and reusable</a></li>
<li>Blog post with examples: <a href="https://gdstechnology.blog.gov.uk/2018/02/19/how-to-open-up-closed-code/">How to open up closed code</a></li>
<li>GOV.UK guidelines for <a href="https://github.com/alphagov/styleguides/blob/master/licensing.md">licensing GDS code</a></li>
<li><a href="http://gds-operations.github.io/guidelines/">GDS open source guidelines</a> for open code that GDS explicitly intends to support</li>
</ul>
<h2 id="security-when-coding-in-the-open">Security when coding in the open</h2>
<ul>
<li>Blog post on GDS technology blog: <a href="https://gdstechnology.blog.gov.uk/2017/09/27/dont-be-afraid-to-code-in-the-open-heres-how-to-do-it-securely/">Don’t be afraid to code in the open: here’s how to do it securely</a></li>
<li>Guidance: <a href="https://www.gov.uk/government/publications/open-source-guidance/when-code-should-be-open-or-closed">When code should be open or closed</a></li>
<li>Guidance: <a href="https://www.gov.uk/government/publications/open-source-guidance/security-considerations-when-coding-in-the-open">Security considerations when coding in the open</a></li>
</ul>
<p>From the security perspective, it’s also worth knowing that while
<a href="https://en.wikipedia.org/wiki/Government_Communications_Headquarters">GCHQ</a> don’t code in the open, they have
released quite a bit of <a href="https://github.com/gchq/">open source code</a>.</p>
<h2 id="gds-open-code">GDS open code</h2>
<ul>
<li>GDS’s GitHub organisation: <a href="https://github.com/alphagov">Alphagov</a></li>
<li>GOV.UK have documented all of the GOV.UK code: <a href="https://docs.publishing.service.gov.uk/">Developer docs</a></li>
<li><a href="https://github.com/alphagov/?utf8=%E2%9C%93&q=verify&type=&language=">GOV.UK Verify code on GitHub</a></li>
<li>Curated list of <a href="https://github.com/nickcolley/awesome-govuk-frontend">GOV.UK Frontend code and
ecosystem</a></li>
<li><a href="https://gdstechnology.blog.gov.uk/category/open-source/">All posts about open code on the GDS technology blog</a></li>
</ul>
<h2 id="coding-in-the-open-across-government">Coding in the open across government</h2>
<ul>
<li><a href="https://government.github.com/community/#uk-central">List of UK central government code on GitHub</a></li>
<li>Blog post by Ministry of Justice: <a href="https://mojdigital.blog.gov.uk/2017/02/21/why-we-code-in-the-open/">Why we code in the open</a></li>
<li>Blog post by DWP: <a href="https://designnotes.blog.gov.uk/2017/03/24/doing-the-hard-work-to-make-things-open/">Doing the hard work to make things open</a></li>
<li>GCHQ’s most popular open source project: <a href="https://github.com/gchq/CyberChef">CyberChef</a></li>
</ul>
<h2 id="let-me-know-if-theres-anything-else">Let me know if there’s anything else</h2>
<p>I hope you find this list useful. I’ll update it as things change so let me know if there’s anything you’d like to see here.</p>
How to open up closed code2018-02-19T00:00:00+00:00https://www.annashipman.co.uk/jfdi/opening-closed-code.html<p>Every digital service designed within government has to meet the <a href="https://www.gov.uk/service-manual/service-standard">Digital Service Standard</a>. One of the requirements of the standard is that new source code should be <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">made open and published under an open source licence</a>.</p>
<p>There are a few situations where it’s acceptable to <a href="https://www.gov.uk/government/publications/open-source-guidance/when-code-should-be-open-or-closed">keep code closed</a> but in most cases it will need to be open.</p>
<p>The easiest way to make code public is to write the code in the open from the beginning, and I’ve <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">previously blogged about the benefits of doing this</a>.</p>
<p>Your team may have old closed code that it needs to open. If there is a lot of closed code, this can be challenging. Here are 3 ways to open it up.</p>
<h2><strong>Option 1: Cycle all credentials then open it</strong></h2>
<p>The commit history is a <a href="https://mislav.net/2014/02/hidden-documentation/">very useful part of a codebase</a> as it explains the reasoning behind changes. If your team wants to maintain the commit history a good approach is to make sure your code contains no secrets and then make it public.</p>
<p>GOV.UK’s infrastructure team did this when it <a href="https://github.com/alphagov/govuk-puppet">opened the GOV.UK Puppet code</a>. The team reviewed the code closely and made sure that any credentials mentioned in the code were no longer in use. They were then able to open up the code. <a href="https://gdstechnology.blog.gov.uk/2016/01/19/opening-gov-uks-puppet-repository/">You can read the infrastructure team’s blog post about it</a> to learn more about what tools and techniques they used to make sure the code was safe to open.</p>
<p>The advantage of this approach is that you maintain the full commit history.</p>
<h2><strong>Option 2: Rewrite history</strong></h2>
<p>If your team feels that the commit history is not suitable for publication, there is an alternative. After reviewing the code to make sure it’s safe to open, you can either rewrite history to remove or improve some commits, or <a href="https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History">squash</a> all previous commits into one. </p>
<p>The MOJ’s Prison Visit Booking team took this approach when it moved from coding privately to coding in the open. The team <a href="https://github.com/ministryofjustice/prison-visits/commit/fc5c908a75cf161b6f3523b5816b76ad3d4adf41">opened the existing code in a snapshot commit</a> (this was the first commit) and then <a href="https://github.com/ministryofjustice/prison-visits">carried on coding in the open</a>.</p>
<p>The advantage of this approach is that you don’t have to touch your infrastructure to do things like change a password or other details. However, the disadvantage is that you lose useful commit history, which can provide context for people working on that codebase. </p>
<h2><strong>Option 3: Move code to a new repo as you use it</strong></h2>
<p>Another way to open closed code is to create a new repository and move code. GOV.UK used this method when <a href="https://gdstechnology.blog.gov.uk/2018/01/05/how-making-our-deployment-code-open-improved-our-workflow/">improving its deployment workflow</a>. This helped the GOV.UK team to spread out the work over a longer time and share the workload out between a larger group of people.</p>
<p>The advantage of this approach is that you can do the work in smaller chunks. And, all the code is examined as you make it open. The disadvantage is that it can take a long time. You’ll also need to make sure your team finishes the job, otherwise you’ll end up maintaining two repos alongside each other.</p>
<h2><strong>After the code is open, work on the open code</strong></h2>
<p>You may be tempted to open code in a snapshot, continue to work in private, and regularly release code to the open version. This is a bad idea because:</p>
<ul>
<li>you need to continue to maintain 2 repos alongside each other, which is extra overhead for your team</li>
<li>you may find it hard to keep up the commitment to open code</li>
<li>you are not coding in the open so you’ll lose the main benefit of <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">encouraging good practice</a>, as well as other benefits</li>
</ul>
<p>I hope these examples will help you find the method that will work best for your team, so you can enjoy all the benefits of coding in the open.</p>
<p><em>This post originally appeared on the <a href="https://gdstechnology.blog.gov.uk/2018/02/19/how-to-open-up-closed-code/">GDS technology blog</a></em>.</p>
A year in the life of an Open Source Lead2017-12-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/a-year-in-the-life-os-lead.html<p>I have been the Open Source Lead at GDS for <a href="https://governmenttechnology.blog.gov.uk/2016/11/18/welcome-to-our-new-open-source-lead/">a year now</a>. Here are some of the things I’ve achieved and learned.</p>
<h2 id="setting-and-implementing-strategy">Setting and implementing strategy</h2>
<p>Open source in government covers a huge range of activities, including open sourcing our own code, using open source software and contributing to open source software. My first task was to define the strategy, work out what areas would have the most impact and then prioritise within those areas. I wrote about the strategy, and the areas I prioritised, <a href="https://governmenttechnology.blog.gov.uk/2016/12/15/next-steps-for-open-source-in-government/">on the GDS blog</a>.</p>
<p>I had to be very focused to stick to these priorities, as lots of people had many interesting ideas for other things I could do. Most of these ideas were excellent suggestions that were just not as high a priority, though I was able to do some of this work as well without getting diverted.</p>
<h2 id="driving-coding-in-the-open">Driving coding in the open</h2>
<p>The UK government has committed to making <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">all new source code open</a>.</p>
<p>Many teams are doing this really well (you can see a lot of government code on the <a href="https://government.github.com/community/#uk-central">GitHub government page</a>). However, some teams find it harder to meet this commitment, so one of my main priorities was helping those teams to overcome the barriers.</p>
<p>The first step to coding in the open is very small; you don’t need to jump in by opening all the code that exists in your organisation. The first thing can be just creating a repository on GitHub with a licence and a README. The step itself is easy; what is difficult is making the decision to take that step. There are many barries, including organisational barriers and emotional barriers.</p>
<p>This year I’ve approached that from three main angles.</p>
<p>Firstly, I’ve shown the value of coding in the open; why it’s worth taking that step. I’ve had many one-to-one chats and given presentations to organisations, and to reach a wider audience I’ve given two conference talks, at <a href="https://www.youtube.com/watch?v=h8vlLRZxedg&feature=youtu.be">GOTO Berlin</a> and <a href="https://www.turingfest.com/speakers/anna-shipman/?wvideo=0p810lpdqr">Turing Fest</a>.</p>
<p>Most importantly, I’ve <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">blogged about the benefits on the GDS blog</a>, which I’ve had feedback has been of huge help to open-source advocates across both government and industry.</p>
<p>Secondly, I’ve addressed the reservations teams have around the practicalities of taking that step. The first question most people have is whether coding in the open is secure. It is. To make that clear, and to share the safeguards you need, I’ve published guidance about <a href="https://www.gov.uk/government/publications/open-source-guidance/when-code-should-be-open-or-closed">when code should be open or closed</a> and <a href="https://www.gov.uk/government/publications/open-source-guidance/security-considerations-when-coding-in-the-open">security considerations when coding in the open</a>. This not only helps people with the details, it also demonstrates our commitment to the policy.</p>
<p>Thirdly, I’ve addressed some of the other practical details. For example, here is some guidance I wrote about <a href="https://github.com/alphagov/styleguides/blob/master/licensing.md">how we license code at GDS</a>.</p>
<h2 id="opening-existing-government-code">Opening existing government code</h2>
<p>As well as helping teams code in the open, I’ve also supported teams opening existing code that had been closed. As well as lots of smaller services, I’m really proud to have supported the opening of three high-profile GDS projects this year. <a href="https://github.com/alphagov/?utf8=%E2%9C%93&q=verify&type=&language=">GOV.UK Verify</a>, <a href="https://govukpay-docs.cloudapps.digital/#contribute">GOV.UK Pay</a> and the <a href="https://github.com/alphagov/ier-frontend">Register to Vote frontend</a>.</p>
<p>The Open Source Lead role is about strategic influence, and the versatility required to persuade and support such varied teams, working with everyone from developers to very senior executives, has been one of the most interesting parts of the role.</p>
<p>My work here has ranged from writing papers for completely non-technical audiences explaining why (and how) the code should be opened, through hands-on help with the code, to influencing senior stakeholders to push for quicker progress from their angle.</p>
<h2 id="building-a-community">Building a community</h2>
<p>One of the main ways to find out about useful code you can learn from and reuse is by hearing people talk about it. It’s also really useful to have people you can discuss shared problems with, so one of the most important things to do was to build the kind of environment where these conversations can happen.</p>
<p>To facilitate this I organised a series of cross-government open source meetups. They were very well attended, each with around 100 participants from 20+ government organisations, and they received 80% <a href="http://blog.verint.com/customer-engagement/net-promoter-score-nps-criticisms-and-best-practices">Net Promoter Scores</a>. <a href="https://twitter.com/TechPunk316">Khidr Suleman</a> did a <a href="https://gdstechnology.blog.gov.uk/2017/10/10/open-source-security-meetup-7-things-we-learned-from-the-cross-government-event/">good write-up of the second meetup</a>.</p>
<p>More important than the events themselves was the community I built around this work. We have a cross-government Slack organisation and there is an <code class="language-plaintext highlighter-rouge">#open-code</code> channel. When I started this role a year ago, this channel was very quiet, and I was involved in most of the conversations. Now I often log into Slack and see that several questions have been asked and answered by other people, and discussions have taken place that haven’t needed my input. This is really heartening to see, as it means the community is supporting each other without my active involvement being required on a daily basis.</p>
<h2 id="learning-and-sharing-how-to-write-business-cases">Learning and sharing how to write business cases</h2>
<p>One thing I had to do a lot more of this year was write business cases, and I wrote <a href="https://www.annashipman.co.uk/jfdi/writing-a-business-case.html">some guidelines</a> on how to do it. I was really pleased when the head of GDS’s delivery management community sent the post round to his team, to show why writing business cases need not be laborious or confusing.</p>
<p>The most interesting piece of work that I wrote a business case for was a discovery into how we could promote reuse of open government code. One addendum is that I followed my own guidelines to write the business case, but although it was well-received (the report I got was that it was “super”) that wasn’t enough to get it approved; I had to keep pushing, and eventually rewrote it as a one-pager, which got a very quick result. In future I’ll try and keep such pitches to one page.</p>
<h2 id="keeping-my-eyes-on-the-prizes">Keeping my eyes on the prize(s)</h2>
<p>Another interesting aspect of roles at this level is that things can take a very long time to come to fruition. I had to demonstrate single-mindedness to see some of them through.</p>
<p>For example, the security guidance I published had a longer than expected road to publication, firstly because of the unexpected general election and then because of a new process around guidance, and it involved liaising with a wide range of stakeholders including <a href="https://www.ncsc.gov.uk/">NCSC</a> and other government departments. I started work on it in January and wasn’t able to publish it until September; but it was worth waiting for: the <a href="https://gdstechnology.blog.gov.uk/2017/09/27/dont-be-afraid-to-code-in-the-open-heres-how-to-do-it-securely/">blog post I wrote announcing it</a> made it to #2 on Hacker News and had 11,000 views in the first 24 hours. More importantly, lots of people have given me feedback that it really helped them with opening their code.</p>
<h2 id="some-other-things-id-like-to-mention">Some other things I’d like to mention</h2>
<p>Alongside these larger themes I did some smaller things I’m pleased with.</p>
<p>I worked with several departments to review and assist with their open source policies. One of the things I’m really proud of is that, based on my feedback, <a href="https://www.gov.uk/government/organisations/hm-courts-and-tribunals-service">HMCTS</a> changed their open source policy from “closed by default” to “open by default” and their technical architect later gave an interesting talk about how that cultural change had happened at our <a href="https://gdstechnology.blog.gov.uk/2017/10/10/open-source-security-meetup-7-things-we-learned-from-the-cross-government-event/">meetup</a>.</p>
<p>I arranged a workshop on contributing to humanitarian open source software; two core committers from <a href="http://openmrs.org/">OpenMRS</a> gave us an introduction to the project and the code, and then we worked on some tickets. We all managed to contribute some code to production during the workshop, which was great.</p>
<p>I’ve supported and encouraged several others to write blog posts about their open source work. Many have been published on the <a href="https://gdstechnology.blog.gov.uk/category/open-source/">GDS technology blog</a> as well as an excellent one from <a href="https://mojdigital.blog.gov.uk/2017/02/21/why-we-code-in-the-open/">the Ministry of Justice</a>.</p>
<p>I’ve also talked to a lot of my counterparts in other governments, and I even found myself writing some lines to send back to an MP responding to a constituent’s inquiry about open source.</p>
<p>I also did a lot of other work moving things along, but can’t yet report the results. That’s one of the interesting things about this level of work, the achievements are larger but the intervening steps tend to be things that are not interesting to talk about (“I had a chat with this person and now have an idea of what to say to that person to convince them that this other course of action might be a good bet…”).</p>
<h2 id="making-open-source-in-government-self-sustaining">Making open source in government self-sustaining</h2>
<p>My aim in any job I do is to ultimately <a href="http://smarterware.org/2010/05/hilary-mason-how-to-replace-yourself-with-a-small-shell-script/">replace myself with a small shell script</a>. For example, a major success this year was when my colleague <a href="https://twitter.com/jenny_duckett">Jenny Duckett</a> was able to help a team through the whole process of opening their code, using guidance I’d written and with almost no input from me.</p>
<p>As Open Source Lead I’ve received a lot of very similar questions on a lot of very similar topics, and much of the work I’ve done this year has been to stop people having to ask those questions by publishing the answers and then publicising them. So my aim in this role is perhaps not to replace myself with a shell script, but rather series of blog posts, guidance, videos, and a shared culture and community. And from the evidence so far, I’ve made significant inroads to that!</p>
Coding in the open in government: Turing Fest talk2017-11-30T00:00:00+00:00https://www.annashipman.co.uk/jfdi/turing-fest.html<p>In August I gave a talk about coding in the open in government at Turing Fest. The video has just been published and you can <a href="https://www.turingfest.com/speakers/anna-shipman/?wvideo=0p810lpdqr">watch it on their site</a> (you can skip the email request).</p>
<p><a href="https://www.turingfest.com/speakers/anna-shipman/?wvideo=0p810lpdqr"><img src="https://embedwistia-a.akamaihd.net/deliveries/b9374a4651032f0cd2ea714c9893788a97a1a192.jpg?image_play_button_size=2x&image_crop_resized=960x540&image_play_button=1&image_play_button_color=54bbffe0" width="400" height="225" style="width: 400px; height: 225px;" /></a></p>
<p>The slides are here:</p>
<div class="embedded">
<iframe src="//www.slideshare.net/slideshow/embed_code/key/32xR6MABgdKd3m" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/annashipman/coding-in-the-open-in-government-78530680" title="Coding in the open in government" target="_blank">Coding in the open in government</a> </strong> from <strong><a href="https://www.slideshare.net/annashipman" target="_blank">annashipman</a></strong> </div>
</div>
<p>If you’d like more detail on any of the things I reference, links are below:</p>
<h2 id="gds-code">GDS code</h2>
<p><a href="https://github.com/alphagov">https://github.com/alphagov</a></p>
<h2 id="scottish-government-code">Scottish government code</h2>
<p><a href="https://github.com/scottishgovernment">https://github.com/scottishgovernment</a></p>
<h2 id="some-other-central-government-code">Some other central government code</h2>
<p><a href="https://government.github.com/community/#uk-central">https://government.github.com/community/#uk-central</a></p>
<h2 id="government-blog-posts-that-i-discussed">Government blog posts that I discussed</h2>
<p><a href="https://gds.blog.gov.uk/2015/09/08/building-a-platform-to-host-digital-services/">Building a platform to host digital services</a></p>
<p><a href="https://gdstechnology.blog.gov.uk/2017/07/18/coding-in-the-open-makes-better-code/">Coding in the open makes better code</a></p>
<p><a href="https://sfadigital.blog.gov.uk/2016/11/17/when-build-a-thing-really-works/">When ‘build a thing’ really works</a></p>
<h2 id="digital-service-standard">Digital service standard</h2>
<p><a href="https://www.gov.uk/service-manual/service-standard">https://www.gov.uk/service-manual/service-standard</a></p>
<h2 id="make-all-new-source-code-open">Make all new source code open</h2>
<p><a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open</a></p>
<h2 id="govuk-coding-styleguides">GOV.UK coding styleguides</h2>
<p><a href="https://github.com/alphagov/styleguides/">https://github.com/alphagov/styleguides/</a></p>
<h2 id="commit-message-guidance">Commit message guidance</h2>
<p><a href="https://github.com/alphagov/styleguides/blob/master/git.md">https://github.com/alphagov/styleguides/blob/master/git.md</a></p>
<h2 id="pull-request-guidance-and-blog-post">Pull request guidance and blog post</h2>
<p><a href="https://github.com/alphagov/styleguides/blob/master/pull-requests.md">https://github.com/alphagov/styleguides/blob/master/pull-requests.md</a></p>
<p><a href="https://gdstechnology.blog.gov.uk/2016/09/30/easing-the-process-of-pull-request-reviews/">Easing the process of pull request reviews</a></p>
<h2 id="be-able-to-recover-quickly">Be able to recover quickly</h2>
<p><a href="https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/">MTTR is more important than MTBF (for most types of F)</a></p>
<h2 id="feedbin-is-open-source">Feedbin is open source</h2>
<p><a href="https://feedbin.com/blog/2013/08/27/feedbin-is-open-source/">https://feedbin.com/blog/2013/08/27/feedbin-is-open-source/</a></p>
<h2 id="govuk-roadmap-trello-and-incident-reports">GOV.UK roadmap, Trello and incident reports</h2>
<p><a href="https://app.productplan.com/p/bUmH4fHC0hOivX-E2LYMu2hg9uEhkWp_">GOV.UK roadmap</a></p>
<p><a href="https://trello.com/b/7yWk0jhI/govuk-publishing-platform-tap-support-planning">GOV.UK publishing platform Trello</a></p>
<p><a href="https://insidegovuk.blog.gov.uk/category/incident-reports/">GOV.UK incident reports</a></p>
Sign up to my new mailing list!2017-11-05T00:00:00+00:00https://www.annashipman.co.uk/jfdi/new-mailing-list.html<p>I usually publicise new posts on Twitter, though I also have an <a href="https://www.annashipman.co.uk/atom">Atom feed</a>. However, Twitter relies on people seeing it at the right time, so I’ve set up a mailing list for new blog posts (and possibly very occasional annoucements).</p>
<h2 id="skip-the-detail-and-just-sign-up">Skip the detail and just sign up</h2>
<div id="mc_embed_signup">
<form action="https://annashipman.us17.list-manage.com/subscribe/post?u=c853878c1ba4f3cfef3649e21&id=1d7520de80" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<div class="mc-field-group">
<label for="mce-EMAIL">Email Address </label>
<input type="email" value="" name="EMAIL" class="required email" id="mce-EMAIL" />
</div>
<div class="mc-field-group input-group" style="min-height: 0px;">
<strong>Email Format </strong>
<input type="radio" value="html" name="EMAILTYPE" id="mce-EMAILTYPE-0" checked="" /><label for="mce-EMAILTYPE-0">HTML</label>
<input type="radio" value="text" name="EMAILTYPE" id="mce-EMAILTYPE-1" /><label for="mce-EMAILTYPE-1">Plain text</label>
</div>
<div id="mce-responses" class="clear">
<div class="response" id="mce-error-response" style="display:none"></div>
<div class="response" id="mce-success-response" style="display:none"></div>
</div>
<!-- anti-bot signup code-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_c853878c1ba4f3cfef3649e21_1d7520de80" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<h2 id="why-ive-set-up-a-mailing-list">Why I’ve set up a mailing list</h2>
<p>Only publicising posts via Twitter relies on you seeing a tweet to know there’s a new post. Twitter is also <a href="https://www.theguardian.com/commentisfree/2017/jan/03/ive-left-twitter-unusable-anyone-but-trolls-robots-dictators-lindy-west">problematic</a>, and it seems like a good idea <a href="https://medium.com/the-mission/want-the-ultimate-career-asset-and-most-durable-form-of-power-start-building-your-platform-fb02ea7bdb1e">not to rely on it</a> as my only way of sharing my writing.</p>
<h2 id="why-you-should-sign-up-to-my-mailing-list">Why you should sign up to my mailing list</h2>
<p>I blog about a bunch of interesting topics, from <a href="https://www.annashipman.co.uk/jfdi/benefits-of-coding-in-the-open.html">open source in government</a> through <a href="https://www.annashipman.co.uk/jfdi/good-pull-requests.html">how to write a good pull request</a> to <a href="https://www.annashipman.co.uk/jfdi/break-into-public-speaking.html">tips on conference speaking</a>, via <a href="https://www.annashipman.co.uk/jfdi/removing-mediawiki-cool-uris.html">more than you ever wanted to know about redirecting URLs</a>, so signing up for the mailing list means you won’t miss out on any of that good stuff.</p>
<h2 id="how-i-set-it-up">How I set it up</h2>
<p>MailChimp currently have this market sewn up; I checked every newsletter I’m subscribed to (I highly recommend <a href="https://www.sandimetz.com/subscribe/">Sandi Metz’s Chainline</a> and <a href="http://ben-evans.com/newsletter/">Benedict Evans’s newsletter</a>, by the way) and they are all powered by MailChimp. The plan is free for the first 2,000 subscribers; after that it becomes relatively pricey and I don’t have a business model for this blog. However, you can extract your mailing list and I figured JFDI; it will take some time to get that many subscribers!</p>
<p>However, I had a lot of issues setting it up, and after the amount of time it took me to create the email template I somewhat wished I’d gone a different route. So if anyone has any suggestions of better mailing list software then do let me know.</p>
<h2 id="a-few-issues-with-mailchimp">A few issues with MailChimp</h2>
<ol>
<li>
<p>The unsubscribe link in the email takes you to a form that requires you to write the email address you want to remove. This is an annoying pattern (we know what it is! You just followed a link from the very email we sent!) and was nearly a deal-breaker for me, but time is limited and I will work out how to migrate away/fix that later.</p>
</li>
<li>
<p>There were some hidden rules about what email addresses are allowed for the mailing list, for example <code class="language-plaintext highlighter-rouge">list@</code> wasn’t allowed. I don’t have a catchall for my domain, so I had to set up a new forwarding address each time without a hint as to which other addresses wouldn’t be allowed.</p>
</li>
<li>
<p>I found the experience of setting up the mailing campaign frustrating; drag-and-drop editing the design of an HTML email must be challenging even if you’re not by default a Vim user. There is also a limit to how many test emails you can send which I didn’t find out about until I’d hit it (it’s 12 for free accounts) and there doesn’t seem to be any way to test a plain-text version of the HTML email — so if you do sign up to receive emails in plain text and it looks rubbish, please do let me know.</p>
</li>
</ol>
<p>It’s also worth noting that they’ve <a href="https://kb.mailchimp.com/lists/signup-forms/single-opt-in-vs.-double-opt-in">changed their default sign-up to not request confirmation</a>.</p>
<h2 id="sending-blog-posts-automatically-is-easy">Sending blog posts automatically is easy</h2>
<p>Having said that, it is free, and it offers some great advantages. It’s very easy to manage subscribers and unsubscribers.</p>
<p>The automation of emailing when a new blog is published via your RSS (or Atom) feed is great and straightforward to set up, it was just designing the email that was taxing. And I’m sure that there are a lot of other advantages to MailChimp that I’ll see when I start using it a bit more.</p>
<h2 id="however-do-let-me-know-if-you-have-other-good-solutions">However, do let me know if you have other good solutions</h2>
<p>There are <a href="http://www.thatsjournal.com/email-marketing/list-of-best-free-open-source-email-list-management-software">some free/open source tools</a> which I didn’t have time to look into, and if anyone has any recommendations I’d love to hear them. I think my requirements are:</p>
<ul>
<li>double opt-in for sign-ups</li>
<li>unsubscribe that doesn’t require you to type your email address again</li>
<li>automate sending blog posts as they are published</li>
</ul>
<p>Also, if you know any MailChimp solutions to my problems above, please let me know.</p>
<p>And do sign up!</p>
An awk command I always forget2017-10-13T00:00:00+00:00https://www.annashipman.co.uk/jfdi/awk.html<p>There’s a task I have to do every now and again for which <code class="language-plaintext highlighter-rouge">awk</code> is the best tool, but it’s infrequent enough that I always have to remind myself how. Usually by referring back to <a href="https://github.com/alphagov/transition-config/blob/7bee2d60f5895893e8db26efa11ef73fc5d42f83/tools/businesslink_prune.sh">some shell scripts we wrote 5 years ago</a>, so thought I’d post here instead.</p>
<h2 id="tell-me-how-many-different-kinds-there-are">Tell me how many different kinds there are</h2>
<p>Given a CSV of people from different government organisations, tell me how many organisations are represented:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>awk -F "," '{ print $4 }' output.csv | sort | uniq -c | wc -l
</code></pre></div></div>
<p>Instead of <code class="language-plaintext highlighter-rouge">wc -l</code>, I usually pipe to a file so I can manually edit out duplicates like MOJ/MoJ/Ministry of Justice, but that’s straightforward once I’ve done all that.</p>
<h2 id="while-im-here-making-notes-of-things-i-forget">While I’m here making notes of things I forget</h2>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find . -iname *utput*
</code></pre></div></div>
Removing MediaWiki from SPA: Cool URIs don't change2017-10-01T10:00:00+00:00https://www.annashipman.co.uk/jfdi/removing-mediawiki-cool-uris.html<p>I run the website for <a href="http://www.spaconference.org">SPA conference</a>. This conference has been running for more than 20 years, and I’ve been the web admin since 2014. This is about one step in updating a 20-year-old legacy PHP site into something more modern: removing the integrated wiki. The story is in two parts and this part is about how I made sure links to the old wiki still worked. It involves many, many things I learned about <code class="language-plaintext highlighter-rouge">.htaccess</code>.</p>
<p>To read the other part of the story, about the actual changes I made to the site, see my previous post: <a href="/jfdi/removing-mediawiki-site-changes.html">Removing MediaWiki from SPA: changes to the site</a>.</p>
<h2 id="getting-rid-of-the-wiki">Getting rid of the wiki</h2>
<p>The integrated wiki was mainly used for session outputs and user pages. I’ve described in <a href="/jfdi/removing-mediawiki-site-changes.html">my previous post</a> how these are now handled on the main site.</p>
<p>When I’d made those changes, there was no longer any need for the wiki. However, there are lots of links to it from previous years, both of outputs and user pages, and <a href="https://www.w3.org/Provider/Style/URI">Cool URIs don’t change</a>.</p>
<p>By far the most time-consuming part of this work was redirecting old links to the new site. I thought it might be less work than my previous <a href="https://gds.blog.gov.uk/2012/12/10/testing-the-redirections/">huge redirection of Direct.gov URLs when we launched GOV.UK</a>, but it was close.</p>
<h2 id="saving-the-wiki-as-a-static-site">Saving the wiki as a static site</h2>
<p>What I wanted to do was save the pages that already existed on the wiki as a static site. So the thing to do was spider the pages and save that. This is something I’m familiar with from <a href="http://devfort.com/">/dev/fort</a>, where we don’t have internet connectivity so we need to take the internet with us.</p>
<p>For a /dev/fort we use <a href="https://github.com/devfort/content-mirror-cookbook">some Chef cookbooks</a> but in this case, I only needed <a href="https://github.com/devfort/content-mirror-cookbook/blob/master/attributes/default.rb#L8">one command</a>:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget -m -k -p -nH -np \
http://www.spaconference.org/mediawiki/index.php?title=Main_Page
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">man wget</code> if you want to know what the arguments are, but to save you a bit of time if you’re only mildly interested:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-m = mirror
-k = convert links
-p = download all page requisites
-nH = Disable generation of host-prefixed directories
-np = no parent; i.e. do not ascend to the parent directory
when retrieving recursively.
</code></pre></div></div>
<p>So then all I had to do was put the output of the spider into a directory called <code class="language-plaintext highlighter-rouge">mediawiki</code>, retire everything that was in the old <code class="language-plaintext highlighter-rouge">mediawiki</code> directory and job done, right? Well, not quite.</p>
<h2 id="the-urls-give-no-information-about-content">The URLs give no information about content</h2>
<p>The first issue was that the web server didn’t know what to do with the files. Most web servers guess at the content type based on the file extension. The files in my static site have names like <code class="language-plaintext highlighter-rouge">http://www.spaconference.org/mediawiki/index.php?title=BelbinTeamRolesInSoftwareDevelopment</code>. Without a file extension, browsers will assume it’s binary.</p>
<p class="half-width"><img src="/img/files_look_like_binary.png" alt="Dialog box indicating server thinks files are binary" /></p>
<p>To solve this, I created a new <code class="language-plaintext highlighter-rouge">.htaccess</code> file in the <code class="language-plaintext highlighter-rouge">mediawiki</code> directory. I wanted to contain strange config needed for retaining the wiki information to the directory itself, rather than messing with the already somewhat complex <code class="language-plaintext highlighter-rouge">.htaccess</code> for the whole site.</p>
<p>I also took this opportunity to move the new MediaWiki folder into a <a href="https://github.com/spaconference/mediawiki_spa">public repo on GitHub</a>. The rest of the SPA code is currently private (which is a story for another day) but there is no reason for this to be, and there is no reason for it to be part of the main SPA repo.
The <a href="https://github.com/annashipman/mediawiki_spa/commit/8faf8c9">first change to the <code class="language-plaintext highlighter-rouge">.htaccess</code></a> was to use <a href="https://httpd.apache.org/docs/current/mod/core.html#filesmatch">Apache’s <code class="language-plaintext highlighter-rouge">FilesMatch</code> directive</a> to tell the server that pages where the URL contains <code class="language-plaintext highlighter-rouge">.php?</code> should be returned as HTML.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code><FilesMatch "\.php\?">
Header set Content-Type "text/html"
</FilesMatch>
</code></pre></div></div>
<h2 id="-has-a-reserved-meaning">? has a reserved meaning</h2>
<p>The next issue was that all the saved MediaWiki file have names like <code class="language-plaintext highlighter-rouge">index.php?title=Main_Page</code>. But <code class="language-plaintext highlighter-rouge">?</code> has a reserved meaning in the browser, to indicate a query string. So the browser will not be looking for a file with a name with a <code class="language-plaintext highlighter-rouge">?</code> in it, it will be looking for what to do with a path of <code class="language-plaintext highlighter-rouge">index.php</code> and a query string of <code class="language-plaintext highlighter-rouge">title=Main_Page</code>.</p>
<p>So I needed to use Apache <a href="https://httpd.apache.org/docs/trunk/mod/mod_rewrite.html">mod_rewrite</a> to rewrite URLs so that <code class="language-plaintext highlighter-rouge">index.php?</code> would be replaced as <code class="language-plaintext highlighter-rouge">index.php%3F</code>. In that case, instead of trying to interpret the query string, Apache would look for the file.</p>
<p>This special requirement took me an extremely long time to work out how to do.</p>
<p>More detail of what it all means in the <a href="https://github.com/annashipman/mediawiki_spa/commit/93ccd2be">commit message</a> but the answer was:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteCond %{QUERY_STRING} (.+)
RewriteRule ^(index\.php)$ $1\%3F%1 [L]
</code></pre></div></div>
<h2 id="spending-ages-on-css-and-javascript">Spending ages on CSS and JavaScript</h2>
<p>I’d redirected the PHP pages in the two stages described above, first making the browser recognise the files were HTML, and then redirecting to the correct filename. So I tried to do the same with the JavaScript and CSS. I initially tried to add a <code class="language-plaintext highlighter-rouge">FilesMatch</code> directive to CSS files but it didn’t have the desired effect. I then tried with the JavaScript but got the same result. It kept saying that the content-type was <code class="language-plaintext highlighter-rouge">text/html</code>.</p>
<p><img src="/img/text_html.png" alt="Image showing JS and CSS as `text/html`" /></p>
<p>This also took me a long time to work out, but I eventually realised that this was because the redirect wasn’t in place – after all, it didn’t have <code class="language-plaintext highlighter-rouge">index.php</code> in the title so it wasn’t getting rewritten. The <code class="language-plaintext highlighter-rouge">text/html</code> was coming from Apache’s 300 (Multiple Choices) response, which was, of course, HTML.</p>
<p><img src="/img/response_html_not_js.png" alt="Image showing the response as a 300 and text/html" /></p>
<p>When I did <a href="https://github.com/annashipman/mediawiki_spa/commit/dcf3615d">both the redirect and the FilesMatch</a> <a href="https://github.com/annashipman/mediawiki_spa/commit/d9716c3b4">at the same time</a> it had the desired effect.</p>
<p>There was an extra JavaScript file with a URL that would have required a whole new level of regexing – <code class="language-plaintext highlighter-rouge">/index.php?title=-&action=raw&gen=js&useskin=monobook&270</code> – but the lack of this file doesn’t appear to have any observable effect apart from a console error so I have left it.</p>
<p>I could also probably tidy up the regexes so there isn’t one rewrite rule for each different file, but there doesn’t seem to be much point – it would only make it slightly harder to understand and introduce the possibility for error. It is not easy to understand <code class="language-plaintext highlighter-rouge">.htaccess</code> files as it is. If I planned for this one to get very long then it might be worth doing, but in this case it’s a finite piece of work: I don’t intend to be updating this directory once this move is finished.</p>
<h2 id="the-spider-hadnt-grabbed-everything">The spider hadn’t grabbed everything</h2>
<p>It was at this point that I discovered that <code class="language-plaintext highlighter-rouge">wget</code> had not grabbed all the files.
Earlier this year I had discovered that redirects had not been set up for one of the previous wikis (there were at least two before MediaWiki, to my knowledge). I had put a lot of redirects in place, and that meant I had a lot of test data that I could use to check links were working. But many were not.</p>
<p>I realised that <code class="language-plaintext highlighter-rouge">wget</code> had not downloaded all of the files because some were not linked to from within the wiki, so <code class="language-plaintext highlighter-rouge">wget</code> didn’t know about them. This was mainly user files, because the wiki didn’t know about all users; it was only the <code class="language-plaintext highlighter-rouge">people.php</code> script, which generated the ‘card index’ that knew about all users (more on that later).</p>
<p><img src="/img/people_page.png" alt="Image showing the 'card index' list of all users" /></p>
<p>I could have crawled that page, but the point of retiring this content is not so much to back up all the data that was ever on the wiki (especially as many of the user pages were blank so were just their name and the wiki chrome), instead it is to preserve old links. So the user pages that I needed were ones that were linked to from previous programmes.</p>
<p>So I put the wiki back in place and crawled it again, this time starting from each of the programme pages between 2011 and 2016 (the years where MediaWiki was in operation and the programme linked out to users).</p>
<p>The command was almost the same as the above, except without the <code class="language-plaintext highlighter-rouge">np</code> flag – no parent. In this case, we did want the spider to go up to the parent directory so that it could then go across to the mediawiki directory to get the users.</p>
<p>This did mean that the spider crawled a lot more than we needed because it was grabbing everything linked to. However, it’s possible to put all the starting directories in one command, in which case <code class="language-plaintext highlighter-rouge">wget</code> doesn’t download the same resource twice. It’s also possible to avoid that by doing <code class="language-plaintext highlighter-rouge">wget</code> into the root of the site locally – whichever folder structure you’re in it builds entire folder structure from root, but I was doing it into a separate directory. I also used <a href="https://itunes.apple.com/gb/app/amphetamine/id937984704">Amphetamine</a> to stop my computer going to sleep, and directed STDERR to a file so I could check afterwards for 500s, 404s, etc.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget -m -k -p -nH \
http://spaconference.org/spa2016/programme.html \
http://spaconference.org/spa2015/programme.html \
[...etc] 2> file.txt
</code></pre></div></div>
<p>I then copied the new output over the existing mediawiki directory and <a href="https://github.com/annashipman/mediawiki_spa/commit/60a00930e">committed only the new files</a>. There was no need to commit any changed files because the changes were just things like timestamps.</p>
<h2 id="redirecting-all-user-pages">Redirecting all user pages</h2>
<p>At this point, the static site contained all the pages from the wiki, and I just needed to create the redirects. Links from previous conferences to users are of the form <code class="language-plaintext highlighter-rouge">http://spaconference.org/scripts/people.php?username={$USER}</code> which redirected the user to the person’s wiki page, i.e. <code class="language-plaintext highlighter-rouge">http://spaconference.org/mediawiki/index.php?title=User:{$username}</code>. So the in order to be able to remove the <code class="language-plaintext highlighter-rouge">people.php</code> script, I just needed to put the redirect in the <code class="language-plaintext highlighter-rouge">.htaccess</code>:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteCond %{QUERY_STRING} ^username\=(.+)$
RewriteRule ^scripts/people.php/?$ /mediawiki/index.php?title=User:%1 [R,L]
</code></pre></div></div>
<p>I then needed to make sure all the redirects worked.</p>
<p>I also needed to deal with the case where the page wasn’t there.</p>
<h2 id="dealing-with-missing-pages">Dealing with missing pages</h2>
<p>On the wiki, where there wasn’t a user you would get:</p>
<p><img src="/img/wiki_empty_user.png" alt="Wiki page for a non-existent user" /></p>
<p>Not ideal, but now there is no MediaWiki framework around it, for a missing page it was even worse:</p>
<p><img src="/img/apache_empty_user.png" alt="Image showing Apache's unstyled 300 reponse page showing multiple possible file matches" /></p>
<p>If there’s no page, I definitely don’t want that 300 result, I want a 404. But it took me a long time to figure out what the issue was. I went down a rabbithole about <a href="https://www.webmasterworld.com/google/3210610.htm">content negotiation</a> and <a href="https://stackoverflow.com/questions/25423141/what-exactly-does-the-the-multiviews-options-in-htaccess">turning off MultiView</a> without success, and then eventually by chance discovered that the second part of the error message (“However, we found documents with names similar to the one you requested”) also appears when Apache is attempting to use the hilariously named <a href="https://httpd.apache.org/docs/current/mod/mod_speling.html">mod_speling</a>.</p>
<p>Check spelling is useful for the main site because it allows URLs like <a href="http://spaconference.org/spa2018/organiser.html">http://spaconference.org/spa2018/organiser.html</a> and <a href="http://spaconference.org/spa2018/Organisers.html">http://spaconference.org/spa2018/Organisers.html</a> to go to the right place.</p>
<p>However, for the wiki site we didn’t want it, so I <a href="https://github.com/annashipman/mediawiki_spa/commit/f15a0820757">turned it off for the mediawiki directory only</a>.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CheckSpelling Off
</code></pre></div></div>
<p>This now returns a 404 if the exact URL can’t be found.</p>
<h2 id="dealing-with-pages-that-had-always-been-missing">Dealing with pages that had always been missing</h2>
<p>There was some extra MediaWiki chrome to deal with. For example, all the pages have an IP address at the top which is apparently a link to a user page for the IP address the user is editing from. However, when I ran the <code class="language-plaintext highlighter-rouge">wget</code> the site was behind CloudFlare, so in fact the IP addresses here are CloudFlare IPs. In any case, the user pages had nothing in them, so a 404 seems appropriate.</p>
<p>Login/create account on any page took you to the <a href="http://www.spaconference.org/scripts/login.php">MySPA login page</a> and then back to the page of the wiki you were on, through a URL something like <code class="language-plaintext highlighter-rouge">http://www.spaconference.org/mediawiki/index.php?title=Special:UserLogin&returnto=User_talk:141.101.99.128</code> and some clever code that linked the wiki to the main conference site. Since it’s no longer possible to log into the wiki (as it is no longer a wiki) I also made this a 404.</p>
<p>I also created a 404 page. Previously, it had been the unstyled Apache 404; not a good look.</p>
<h2 id="linking-to-new-user-pages">Linking to new user pages</h2>
<p>Turning off <code class="language-plaintext highlighter-rouge">CheckSpelling</code> avoided lots of 300 errors, but it also meant that a lot of pages were missing because of the URLs. On the wiki, several of the links had extra query parameters. A really common one was <code class="language-plaintext highlighter-rouge">&useskin=bio</code> because it turned out the PHP pages that created the ‘card index’ added this parameter.</p>
<p>The people pages were linked up to the wiki via <code class="language-plaintext highlighter-rouge">scripts/people.php</code>. For example: <code class="language-plaintext highlighter-rouge">http://spaconference.org/scripts/people.php?username=MarinaHaase</code>. The people script passed that call through to a Smarty template, code below:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code><html>
<head>
<title>SPA Conference - People</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<frameset cols="184,*" border=0>
<frame name="index" src="/people/index.html" marginwidth=0>
<frame name="person" src="/mediawiki/index.php?title=User:{$username}&useskin=bio" marginwidth=0>
</frameset>
<noframes>
<body bgcolor="#FFFFFF" text="#000000">
We're sorry - the SPA people page requires a browser that supports frames.<br>
<br>
<a href="index.html">Click here</a> to return to the conference site.
</body>
</noframes>
</html>
</code></pre></div></div>
<p>The template created a page made up of two frames, one being the ‘card index’, which was actually just a list of all users, and the other frame loading the user’s page from the wiki. In the example above, the frame would display <code class="language-plaintext highlighter-rouge">/mediawiki/index.php?title=User:MarinaHaase&useskin=bio</code>.</p>
<p>(Either that, or it said “We’re sorry - the SPA people page requires a browser that supports frames.”)</p>
<p>An interesting result of that was that if you did go to the wiki via a link like the one above and then choose someone else from the ‘card index’, the URL would remain the same.</p>
<p>However, it turned out that none of the stored files have <code class="language-plaintext highlighter-rouge">&useskin=bio</code> in their title, so this would lead to a 404 even when the page was present.</p>
<p>I solved this by adding a redirect to <a href="https://github.com/annashipman/mediawiki_spa/commit/2ad76858">ignore all second query parameters</a>.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteCond %{QUERY_STRING} (.+)&(.+)
RewriteRule ^(index\.php)$ $1\%3F%1 [L]
</code></pre></div></div>
<p>This rule looks for all query strings that have two parameters and then discards the second one. (We still need the first one because it’s the user’s name, e.g. <code class="language-plaintext highlighter-rouge">?title=User:MarinaHaase</code>.)</p>
<h2 id="removing-all-the-people-files">Removing all the people files</h2>
<p>With all these redirects in place, I could now remove all the PHP files that connected to the wiki, e.g. the Smarty template shown above.</p>
<p>I could also remove all the MediaWiki tables from the database – there were 49 MediaWiki tables in total.</p>
<p>This was extremely satisfying.</p>
<h2 id="that-wasnt-the-end-of-it">That wasn’t the end of it</h2>
<p>Before MediaWiki was the wiki, there were (at least) two other wikis for the SPA site, and when they were replaced, there were redirects to the new site in the <code class="language-plaintext highlighter-rouge">.htaccess</code>. However, they just redirected to <code class="language-plaintext highlighter-rouge">/mediawiki/</code>, relying on <code class="language-plaintext highlighter-rouge">index.php</code> to do the right thing. For example:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteRule ^cgi-bin/wiki.pl/?$ /mediawiki/ [L]
</code></pre></div></div>
<p>However, there isn’t an <code class="language-plaintext highlighter-rouge">index.php</code> any more, just a bunch of flat HTML files.</p>
<p>I updated all those redirects to the main wiki page, for example:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteRule ^scripts/wiki$ /mediawiki/index.php?title=Main_Page [R,L]
</code></pre></div></div>
<p>I needed to add the <a href="https://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_r"><code class="language-plaintext highlighter-rouge">[R]</code> flag</a> to redirect the URL. Without a redirect, the page doesn’t get the CSS or other assets as they are referenced using relative paths.</p>
<h2 id="how-do-you-send-a-403-to-a-404">How do you send a 403 to a 404?</h2>
<p>These changes meant that there were a few more pages where Apache returned a 403 Forbidden. In particular, <code class="language-plaintext highlighter-rouge">http://www.spaconference.org/mediawiki/</code> and <code class="language-plaintext highlighter-rouge">http://spaconference.org/people/</code> both no longer have an index page to do the right thingwhich resulted in a 403: ‘You don’t have permission to access /mediawiki/ on this server.’</p>
<p>Firstly, this was again an unstyled Apache response page, and secondly, it makes it sound more interesting than it is. It’s not that there’s something cool there you can’t see, it’s just “hey everyone! We use Apache!”.</p>
<p>My initial plan was to redirect the 403 to a 404 (This is an acceptable way to handle a 403, according to <a href="https://tools.ietf.org/html/rfc2616#page-66">RFC2616</a>.)
However, I could not work out how. One of the problems with Apache is that it’s very hard to Google for help, as it’s been in use for a very long time, so the internet is full of people asking questions and giving answers, many of whom do not have a clue. Much Googling and RTFMing led me nowhere. In desperation I even tried a suggestion to replace the 403 error document with a 404, which led to the worst of both worlds:</p>
<p><img src="/img/403_and_404.png" alt="Page showing unstyled 403 *and* 404z" /></p>
<p>As an interim measure, I just returned the 404 pages for a 403 response, which is suboptimal but fine from most users’ perspective.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ErrorDocument 403 /404.php
</code></pre></div></div>
<h2 id="what-are-users-trying-to-do">What are users trying to do?</h2>
<p>When users are trying to access <code class="language-plaintext highlighter-rouge">/people/</code>, they are probably trying to find out whether there is a list of all people there. So my first plan here was for <code class="language-plaintext highlighter-rouge">people.html</code> to have some text explaining what had happened. Here is this, unstyled (so with no SPA conference chrome around it).</p>
<p>However, even styled, this would have been no use at all.</p>
<p><img src="/img/unstyled_people_html.png" alt="Page showing some not very useful text about the former existence of the people page" /></p>
<p>In addition, because the disabling of <code class="language-plaintext highlighter-rouge">CheckSpelling</code> is in the <code class="language-plaintext highlighter-rouge">/mediawiki/</code> directory not at the top level, <code class="language-plaintext highlighter-rouge">http://www.spaconference.org/people/</code> is a 300, and the option it offers is <code class="language-plaintext highlighter-rouge">http://www.spaconference.org/people.html/</code>, which is a 404.</p>
<p><img src="/img/people_300.png" alt="Image of `/people/` directory being a 300" /></p>
<p>In summary, this was a terrible idea.</p>
<p>So I reconsidered the 403 to 404 approach and decided to redirect both of <code class="language-plaintext highlighter-rouge">/mediawiki/</code> and <code class="language-plaintext highlighter-rouge">/people/</code> the wiki home page. In the case of <code class="language-plaintext highlighter-rouge">/people/</code> this is not really what they want either, but there is no longer a list of all SPA people and this will give them something.</p>
<p>This was straightforward for <code class="language-plaintext highlighter-rouge">/people/</code>.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteCond %{QUERY_STRING} ^$
RewriteRule ^people/?(\.html)?$ /mediawiki/index.php?title=Main_Page [R,L]
</code></pre></div></div>
<p>In the case of <code class="language-plaintext highlighter-rouge">http://www.spaconference.org/mediawiki/</code> it was slightly more interesting. <code class="language-plaintext highlighter-rouge">/mediawiki</code> to <code class="language-plaintext highlighter-rouge">/mediawiki/</code> was already a 301 because of <code class="language-plaintext highlighter-rouge">CheckSpelling</code>, and if I put an index.html in the mediawiki directory then it would redirect to that, but I couldn’t use a <code class="language-plaintext highlighter-rouge">^$</code> redirect. I eventually figured out that this was because there is actually a directory there.</p>
<p>I found that if I put in a <code class="language-plaintext highlighter-rouge">Redirect</code> rather than a <code class="language-plaintext highlighter-rouge">RewriteRule</code> this worked, e.g.:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Redirect 302 /mediawiki/index.html http://www.spaconference.org/mediawiki/index.php?title=Main_Page`
</code></pre></div></div>
<p>(<code class="language-plaintext highlighter-rouge">index.html</code> doesn’t need to exist for this redirect to work)</p>
<p>However, then I realised that this work shouldn’t be done in that directory and in fact it’s a feature of the <code class="language-plaintext highlighter-rouge">mediawiki</code> folder that it doesn’t have an <code class="language-plaintext highlighter-rouge">index.html</code> and its homepage is actually the oddly titled <code class="language-plaintext highlighter-rouge">index.php?title=Main_Page</code>, so I deleted the <code class="language-plaintext highlighter-rouge">Redirect</code> and did the work <a href="https://github.com/annashipman/mediawiki_spa/commit/fc0fa3191">in the mediawiki .htaccess instead</a>:</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteCond %{QUERY_STRING} ^$
RewriteRule ^$ /mediawiki/index.php?title=Main_Page [R,L]
</code></pre></div></div>
<p>This solves the 403 to 404 issue. But I’m still interested if you can tell me how to redirect 403s to 404s correctly in Apache.</p>
<h2 id="and-that-is-the-end-of-it-i-think">And that is the end of it, I think…</h2>
<p>And I think we’re done. MediaWiki has been removed, but all the old outputs and information are <a href="http://www.spaconference.org/mediawiki/index.php?title=Main_Page">preserved at the same URLs</a>. Please do let me know if you spot any issues.</p>
<p>This was a lot of work but it was really fun and I learned far more about <code class="language-plaintext highlighter-rouge">.htaccess</code> than I’d ever intended to. I’m happy it’s moved me one step closer to bringing the SPA site up to date.</p>
Removing MediaWiki from SPA: Changes to the site2017-10-01T09:00:00+00:00https://www.annashipman.co.uk/jfdi/removing-mediawiki-site-changes.html<p>I run the website for <a href="http://www.spaconference.org">SPA conference</a>. This conference has been running for more than 20 years, and I’ve been the web admin since 2014. This is about one step in updating a 20-year-old legacy PHP site into something more modern: removing the integrated wiki. The story is in two parts, and this part is about why I removed the integrated wiki and the changes I made to the function of the site to accommodate that.</p>
<p>The second part is about <a href="/jfdi/removing-mediawiki-cool-uris.html">how I made sure the wiki pages were still available</a>.</p>
<h2 id="the-site-included-a-discussion-wiki">The site included a ‘discussion wiki’</h2>
<p>In early years, the conference was residential and had a much smaller, close community, and the wiki was used much more.</p>
<p>The wiki was an installation of <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a>, modified so that conference website and the wiki could share a single login.</p>
<p>In recent years, the wiki has been used for two main things: session outputs and user pages.</p>
<h2 id="session-outputs">Session outputs</h2>
<p>The sessions at SPA are mostly interactive, so session outpts can included slides, working code, links to GitHub repos and blog posts, etc.</p>
<p>Session leaders were encouraged to add output from the session to the wiki and you could then <a href="http://www.spaconference.org/mediawiki/index.php?title=SpaTwoThousandAndFourteenOutput">look at all outputs from the conference</a>.</p>
<p>To replace this, I added the ability for conference presenters to update their sessions with the outputs. You can see an example under the heading ‘Outputs’ for the session <a href="http://spaconference.org/spa2017/sessions/session694.html">Access control for REST services</a>.</p>
<h2 id="user-pages">User pages</h2>
<p>If you are submitting a proposal to SPA, you need to <a href="http://www.spaconference.org/scripts/myprofile.php">sign up to the conference site</a>, where you fill in some details and, if you consented, a user page was created on the wiki.</p>
<p>In previous years tickets were also purchased through the site, so this meant all attendees would also sign up to the conference site, and so most attendees would have a wiki page. The wiki (helped by some PHP scripts) showed a ‘card index’ of these user pages to help with the community feel:</p>
<p><img src="/img/people_page.png" alt="Page showing a list of all user pages" /></p>
<p>However, the conference is now hosted by the <a href="http://www.bcs.org/">BCS</a>, who handle ticketing, so new profiles were only created for session leaders and conference committee members. The ‘card index’ wasn’t at all up to date with who you might meet at the conference.</p>
<p>I moved the user pages to the site itself, rather than the wiki. And we don’t need everyone; just speakers.</p>
<p>Before:</p>
<p><img src="/img/user_page_on_wiki.png" alt="A user page on the wiki" /></p>
<p>After:</p>
<p><img src="/img/user_page_on_site.png" alt="The same user page on the site" /></p>
<p>As part of this, I also made a change to the way the site handled images. Previously, when a user uploaded an image, it would be stored by MediaWiki and that would be the image associated with them. A new image would update their user page. Now, user images are uploaded to a directory called mugshots. When a user is a session leader for a particular conference, and they have an image in mugshots their image will be copied over to the current year’s images directory and used in their user page.</p>
<p>Two reasons to copy the image over rather than just linking to mugshots:</p>
<ul>
<li>Each year’s site should be complete in itself and not reply on images from outside of itself</li>
<li>A user may upload an updated photo for a session in a different year; that should not change the photo used in a previous year.</li>
</ul>
<h2 id="updating-profile-page">Updating profile page</h2>
<p>I also made some changes to the profile page at the same time. For example, MediaWiki markup is no longer allowed. It was also collecting a lot of information like address, phone number and fax (!), but now bookings are no longer done through the site, there’s no need to collect this information. I also deleted those columns from the database; there’s no need to store it for existing users.</p>
<p><img src="/img/profile_page_with_consent_options.png" alt="Old profile page showing out of date fields" /></p>
<p>I also checked whether that information was published on the wiki so I could remove it from there, but even though there were users who had checked the ‘publish address’ checkbox, no addresses were actually published on the wiki.</p>
<p>In the near future I’ll make more changes to the user page so that we actually get consent (or not) for marketing, rather than the implied consent seen here (“By registering with this site you are giving consent for the BCS SPA Specialist group to contact you…”); both in order to comply with <a href="https://ico.org.uk/for-organisations/data-protection-reform/overview-of-the-gdpr/">GDPR</a>, but also because that’s a horrible dark pattern.</p>
<p>This was also a good opportunity to change the markup from a table to more modern mark-up that will allow it to be read on a smaller device.</p>
<h2 id="users-without-details">Users without details</h2>
<p>Many of the sessions at SPA are lead by two or more session leaders, and if their co-presenters are not already registered on the site, naming them in the proposal creates an empty record (i.e. it bypasses all the compulsory fields). This leads to user pages with no content. This is what that looks like:</p>
<p>Before:</p>
<p><img src="/img/wiki_empty_user.png" alt="A wiki page showing a user with no details" /></p>
<p>After:</p>
<p><img src="/img/site_empty_user.png" alt="A page on the site showing a user with no details" /></p>
<p>I think the after looks better. However, one potential addition is a script to let the programme chairs know which speakers need to pad out their bios. There’s no need to force this information at the proposal stage, as some proposals will be rejected.</p>
<h2 id="getting-rid-of-the-wiki">Getting rid of the wiki</h2>
<p>Having made these changes, there was no longer any need for the wiki. However, there are lots of links to it from previous years, both of outputs and user pages, and <a href="https://www.w3.org/Provider/Style/URI">Cool URIs don’t change</a>.</p>
<p>This was a lot of work, so I’ve written it up in a separate post. <a href="/jfdi/removing-mediawiki-cool-uris.html">Read on for part 2: cool URIs don’t change</a>.</p>
Don’t be afraid to code in the open: here’s how to do it securely2017-09-27T00:00:00+00:00https://www.annashipman.co.uk/jfdi/coding-open-securely.html<p>There are two big concerns government organisations have around <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">making source code open</a>. They want to know which subsets of the code should be kept closed and how to code in the open securely. To address these questions I’ve introduced two pieces of guidance:</p>
<ul>
<li><a href="https://gov.uk/government/publications/open-source-guidance/when-code-should-be-open-or-closed">'When code should be open or closed'</a></li>
<li><a href="https://gov.uk/government/publications/open-source-guidance/security-considerations-when-coding-in-the-open">'Security considerations when coding in the open'</a></li>
</ul>
<p>Both pieces of guidance are based on industry standards and have been reviewed by the GDS security engineering team as well as government colleagues from National Cyber Security Centre, Department for Work and Pensions, Home Office and Ministry of Justice.</p>
<h2>Why we have updated the guidance</h2>
<p>We previously blogged about what code not to open in our post: '<a href="https://gds.blog.gov.uk/2014/10/08/when-is-it-ok-not-to-open-all-source-code/">When is it ok not to open all source code?</a>', but as guidance, it is no longer relevant as our approach to specific areas such as configuration have changed. For example, last year we made <a href="https://gdstechnology.blog.gov.uk/2016/01/19/opening-gov-uks-puppet-repository/">GOV.UK's Puppet repository</a> publically available on GitHub.<p>
<p>We’ve also <a href="https://gdstechnology.blog.gov.uk/2016/07/13/why-security-says-no-wont-cut-it-anymore/">evolved our thinking on security</a>. The previous guidance discouraged people from sharing code that had anything at all to do with security. It didn’t take into account that coding in the open can actually make code more robust as it helps you design with security in mind.</p>
<p>The new guidance addresses why open sourcing code that performs a security-enforcing function is beneficial. In simple terms, we can compare coding in the open to how padlocks work. Everyone knows how padlocks work but they are still secure because you cannot open them without the key. <a href="https://en.wikipedia.org/wiki/RSA_(cryptosystem)">Security enforcing software works in the same way</a>, and good cryptographic algorithms are reviewed by many professional peers. Security is improved through public review.</p>
<p>We still specifically seek peer review on open code and subject our code to penetration testing, as part of following <a href="https://www.ncsc.gov.uk/guidance/security-design-principles-digital-services-main">security design principles</a>.</p>
<h2>What we’re doing at GDS</h2>
<p>Most of the code produced by GDS has been coded in the open from the beginning. Some services started closed source, and to ensure that we are practicing what we preach, we are now opening those: we have recently completed open sourcing <a href="https://govukpay-docs.cloudapps.digital/#contribute">GOV.UK Pay</a> and we are working on opening up more components of <a href="https://github.com/alphagov/verify-frontend/">GOV.UK Verify</a>. </p>
<p>This new guidance will make it easier for your organisation to develop and deploy secure and open services, and should address your concerns around coding in the open securely.</p>
<p><em>This post originally appeared on the <a href="https://gdstechnology.blog.gov.uk/2017/09/27/dont-be-afraid-to-code-in-the-open-heres-how-to-do-it-securely/">GDS technology blog</a></em>.</p>
</p></p>
The benefits of coding in the open2017-09-04T00:00:00+00:00https://www.annashipman.co.uk/jfdi/benefits-of-coding-in-the-open.html<p>For any service to be put in front of the public, it has to meet the Digital Service Standard, a set of 18 criteria.</p>
<p>One of the criteria is that all new source code is <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">made open and published under an open source licence</a>.</p>
<p>This goes hand in hand with our tenth design principle: <a href="https://www.gov.uk/design-principles#tenth">make things open: it makes things better</a>.</p>
<p>In this blog post, I explain why coding in the open makes things better.</p>
<p><img src="/img/coding-in-the-open.jpg" alt="two developers in front of a screen with code on it" /></p>
<h2><b>It encourages good practice</b></h2>
<p>When you know someone is watching, you tend to take greater care. You're more inclined to document your work clearly. You make sure your code is secure by keeping secrets separate from the code. You are polite and constructive in code reviews, and you follow good architectural principles.</p>
<p>In short: when other people can see your work, you tend to raise your game.</p>
<h2><b>It makes collaboration easier</b></h2>
<p>If code is open, it is easier to work on it with others. You don't need to give them special access or make complicated business arrangements. You don't even need to be in the same building.</p>
<p>For example, someone from 18F, the government agency that provides digital services to the government of the United States, was able to <a href="https://gdstechnology.blog.gov.uk/2017/07/18/coding-in-the-open-makes-better-code/">help a colleague from GDS with a code-writing problem</a>.</p>
<p>It worked because both sides coded in the open. We also <a href="https://governmenttechnology.blog.gov.uk/2016/09/06/celebrating-sharing-and-reusing-the-digital-marketplace/">worked with the Australian Government</a> to help them establish their own Digital Marketplace.</p>
<p>Closer to home, it makes it easier to work on the same code between departments.</p>
<h2><b>External users can help make it better</b></h2>
<p>Open code makes it possible for people who don’t work for you to make improvements to your code.</p>
<p>For example, members of the public made improvements to the <a href="https://petition.parliament.uk/">Government Petitions Service.</a> Someone <a href="https://github.com/alphagov/e-petitions/pull/456">added the scheduled date</a> for debates. Someone else made a change to the signature counter to <a href="https://github.com/alphagov/e-petitions/pull/482">make it update in real time</a>.</p>
<p>People can 'scratch their own itches'. They can make the small improvements that aren't at the top of your list of priorities, and they can help make your code more robust.</p>
<h2><b>Others can learn from your work</b></h2>
<p>If your code is open, people can apply what you've learned from doing the work.</p>
<p>Skills Funding Agency used GOV.UK's Smart Answers code to <a href="https://sfadigital.blog.gov.uk/2016/11/17/when-build-a-thing-really-works/">build a tool for their apprenticeships service</a>. It took less than a week.</p>
<p>Without the Smart Answers example to learn from, it would have taken at least two months.</p>
<h2><b>It makes it easier to share standards</b></h2>
<p>Open code makes it easy to follow other teams’ work. This promotes a common culture and way of working when you can see how other teams manage certain issues.</p>
<p><img src="/img/Anna-Shipman-768x512.jpg" alt="Anna Shipman and another member of GDS staff" /></p>
<p>Quite often, teams will make small improvements to other teams’ work. For example, a developer from GOV.UK <a href="https://github.com/alphagov/verify-frontend/pull/255">made a correction to GOV.UK Verify</a>.</p>
<p>GOV.UK publishes <a href="https://github.com/alphagov/styleguides/">coding style guides.</a> This makes it easy for everyone to find and stick to the same standards.</p>
<h2><b>It improves transparency on government’s work</b></h2>
<p>When code is developed in the open, you can see where public money goes.</p>
<p>It is a catalyst which encourages openness in other things. For example, the <a href="https://insidegovuk.blog.gov.uk/2017/02/13/the-2017-to-2018-gov-uk-roadmap/">GOV.UK roadmap is open</a>, and one of the teams on GOV.UK uses a <a href="https://trello.com/b/7yWk0jhI/govuk-publishing-platform-tap-support-planning">public Trello board</a>.</p>
<p>When there is an occasional outage on GOV.UK we investigate and <a href="https://insidegovuk.blog.gov.uk/category/incident-reports/">publish a report</a>. It’s important to show how we learn from mistakes.</p>
<h2><b>It clarifies ownership</b></h2>
<p>We want government to own and be able to make changes to its services, and lack of clarity on intellectual property (IP) can be a barrier to that.</p>
<p>Open coding from the beginning surfaces copyright and IP issues before work starts.</p>
<p>The Service Standard demands that code is published under an open source licence (<a href="https://github.com/alphagov/styleguides/blob/master/licensing.md">at GDS we use MIT</a>). Additionally, all the work we do as civil servants is Crown copyright.</p>
<p>In the past, government services have wanted to change a project but have been unclear about who owns the IP.</p>
<p>Clarifying the issue upfront is valuable. It means that departments can bring in a supplier to work on their alpha and then switch to another supplier for beta without losing their work.</p>
<p>They can even build up teams from many suppliers who can work on the code seamlessly.</p>
<p>It prevents supplier lock-in. Without clarification, the software created for you can be the thing that will prevent you from switching suppliers.</p>
<p>So resolving this can save a lot of money for government.</p>
<h2><b>It helps make government technology seamless</b></h2>
<p>People who move between departments can continue to work using the same tools as before. It saves time and money. They can share knowledge of projects they were working on, because it’s all open.</p>
<p>After someone moved from GDS to another department, they <a href="https://github.com/alphagov/signon/pull/509">contributed to our single sign-on service</a>.</p>
<p>Over time, it will make government technology seamless as people move towards the most useful tools.</p>
<h2><b>It’s easier to code in the open than to open a closed repository</b></h2>
<p>Coding in the open means you decide whether that code is suitable for publication as part of reviewing each small piece of work.</p>
<p>To open it later means having to go back through a body of work that has built up over time to make sure there is nothing that shouldn’t be made public, which can be significant extra work. </p>
<h2><b>Make your own code open</b></h2>
<p>Many people think that being able to reuse code is the biggest benefit of coding in the open. However, while reuse is a nice-to-have, I hope this blog post illustrates that there’s more to it than that.</p>
<p>Take a look at <a href="https://github.com/alphagov/">our open code</a> and our <a href="https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable">guidance.</a></p>
<p><em>This post originally appeared on the <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">GDS blog</a></em>.</p>
Break into public speaking2017-08-23T00:00:00+00:00https://www.annashipman.co.uk/jfdi/break-into-public-speaking.html<p>Earlier this year I had the opportunity to work with my excellent colleagues <a href="https://twitter.com/rosaemerald">Rosa Fox</a> and <a href="https://twitter.com/lucyrushi">Lucy Carey</a> on a series of workshops to help get more underrepresented people in tech into public speaking. Lucy has written an <a href="https://gds.blog.gov.uk/2017/06/21/breaking-in-a-new-generation-of-public-speakers/">excellent blog post about it</a> including more details about the breakdown of the course.</p>
<p>This is something I’m really interested in. I’ve written before about <a href="/jfdi/how-to-get-women-speakers.html">how to get more women to speak at your conference</a> and shared <a href="/jfdi/conference-speaking.html">resources on getting started with conference speaking</a>, so I was really happy to help with it.</p>
<h2 id="you-definitely-have-something-interesting-to-talk-about">You definitely have something interesting to talk about</h2>
<p>The main thing that I want people to realise is that everyone has something interesting to speak about (or <a href="/jfdi/how-to-blog.html">blog about</a>). <a href="https://twitter.com/jessicaivins">Jessica Ivins</a> has written a <a href="http://cognition.happycog.com/article/so-why-should-I-speak-publicly">great post about this</a>. You know a lot of things that other people don’t know and would find interesting.</p>
<p>The topics that came out of the first workshop were many and varied, including a deeper dive into one aspect of a project someone was working on, advice about how to make it as a project manager, developing junior developers (which <a href="https://twitter.com/EmmaBeynon">Emma Beynon</a> went on to give at <a href="https://brightonruby.com/2017/developing-junior-developers-emma-beynon/">Brighton Ruby</a>), and a day in the life of business support.</p>
<h2 id="you-only-need-to-be-a-few-hours-ahead-of-the-audience">You only need to be a few hours ahead of the audience</h2>
<p>We often think we have to be expert in something in order to give a talk about it, but that’s not the case. Even in something deeply technical, you only need to be about three hours ahead of the audience, and in fact the closer you are to their level of understanding, the clearer your talk can be.</p>
<p>Finally, it’s worth remembering that a deadline of a talk is the best way to make sure you really know a subject. The reason I submitted my first conference workshop, <a href="http://www.spaconference.org/spa2012/sessions/session412.html">Data Visualisations in JavaScript</a> was because I wanted to know how to to do it. It was a lot of work, but I really knew it by the time of the session!</p>
<p>Break into public speaking got lots of interest on the back of Lucy’s post (including international attention!) and the next run of it starts next week. I really look forward to seeing where it goes next.</p>
Cross government meetup on Open Source and Security2017-08-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/security-and-oss-meetup.html<p>We are hosting the second cross-government meetup on Open Source in London on Tuesday 26th September.</p>
<p><img src="/img/Make-things-open.png" alt="Make things open image" /></p>
<p>The event will focus on security considerations around Open Source and talks will be given by:</p>
<ul>
<li>a representative from <a href="https://www.ncsc.gov.uk/">NCSC</a>, who will discuss integrating security into software development from the perspective of the developer</li>
<li>Tom Price, Head of FinTech at <a href="https://www.gov.uk/government/organisations/hm-treasury">HM Treasury</a> (formerly Head of Emerging Sectors at BEIS) who will explore the benefits and risks of blockchain.</li>
<li>Jason Paige from <a href="https://www.gov.uk/government/organisations/hm-courts-and-tribunals-service">HM Courts and Tribunal Service</a> who will talk about his experience of changing the culture of his team from one that didn’t release any code because of “security reasons” to one that is open by default.</li>
</ul>
<p>Following this, there will be a panel of security experts, including Ahana Datta, Head of Technical Security at <a href="https://www.gov.uk/government/organisations/ministry-of-justice">Ministry Of Justice</a>, Jenny Duckett from the GDS security engineering team and a representative from NCSC, who will answer questions from the audience about open source and security.</p>
<p>There will then be time for networking and <a href="https://en.wikipedia.org/wiki/Unconference">open space discussion</a> to explore security in-depth or discuss other topics.</p>
<p>The aim is to give you tools to take back to your organisations to support your work with Open Source. Everyone working in government is welcome, whether you’re against the idea of open source or already an advocate. In fact, if you’re not convinced, we’d be even more keen to have you come, so we can understand and try to address your reservations.</p>
<h2 id="a-recap-of-the-previous-event">A recap of the previous event</h2>
<p>The <a href="https://governmenttechnology.blog.gov.uk/2017/01/25/come-to-the-cross-government-open-source-meetup/">first in this series</a> of cross-government meetups on Open Source was in February and had a strong turnout, with around 75 attendees from over 20 different government organisations. The feedback was generally positive, with over 80% saying they found it useful and would recommend it to a colleague.</p>
<p>We had talks from <a href="https://twitter.com/daverog">Dave Rogers</a>, CTO of MOJ, explaining <a href="https://mojdigital.blog.gov.uk/2017/02/21/why-we-code-in-the-open/">why they code in the open</a>, someone from <a href="https://www.gchq.gov.uk/">GCHQ</a> talking about <a href="https://github.com/gchq">their open source</a> projects, and <a href="https://twitter.com/gemmaleigh">Gemma Leigh</a> talking about the <a href="https://github.com/alphagov/govuk-frontend">GOV.UK Frontend Alpha</a>.</p>
<p>This was followed by 10 Open Space sessions discussing topics suggested by participants. There was some really useful sharing of thoughts and experiences.</p>
<h2 id="giving-people-the-tools-to-help">Giving people the tools to help</h2>
<p>Something I noticed at the event was that in some ways we were preaching to the choir. People who will travel to a meetup about Open Source are people who are already working hard to promote it in their organisations. While an opportunity to discuss common areas of frustration can be cathartic, what would be more useful would be giving people the tools to have those conversations back in their organisations with people who are less enthusiastic about Open Source.</p>
<p>One of the primary concerns people have about using Open Source Software and <a href="https://gdstechnology.blog.gov.uk/2017/07/18/coding-in-the-open-makes-better-code/">coding in the open</a> is the misconception that it is less secure. So this meetup will provide tools, experiences and information to help attendees combat those concerns.</p>
<h2 id="better-opportunities-for-networking">Better opportunities for networking</h2>
<p>A few people commented in their feedback for the last event that the structure of the day didn’t help them get to know the other attendees. I think it’s important to fix this because I want it to be easy for people to network and meet others in the Open Source community if they want to.</p>
<p>At this meetup we will have some structured networking activities so those who want to take part and would like an opportunity to meet others can do so. It will be organised by <a href="https://twitter.com/janeoloughlin">Jane O’Loughlin</a>, who has a lot of experience running meetups and communities. Don’t worry, the networking will not be compulsory!</p>
<h2 id="location-location-location">Location, location, location</h2>
<p>In the original blog post I said that we would host other events elsewhere in the UK but feedback from people coming from outside London is that it’s a convenient location to get to. However, this was based on feedback from people who managed to make it, so if the location is preventing your attendance, please let us know, either in the <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">cross-government Slack channel</a> or in the comments below.</p>
<h2 id="let-us-know-what-else">Let us know what else</h2>
<p>The agenda of the next one is not set in stone so please feel free to share any thoughts or suggestions, for this or future events. And please do come along! Everyone working in government (both civil servants and contractors) is welcome. The invitation will be shared in the <a href="https://ukgovernmentdigital.slack.com/messages/open-code/">cross-government Slack</a> and via <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">mailing lists</a>.</p>
<p><em>This post originally appeared on the <a href="https://gdstechnology.blog.gov.uk/2017/08/08/sign-up-for-our-cross-government-meetup-on-open-source-and-security/">GDS Technology Blog</a>.</em></p>
Join the SPA organising committee2017-06-13T00:00:00+00:00https://www.annashipman.co.uk/jfdi/join-spa-committee.html<p>I help run <a href="http://spaconference.org/">SPA</a>, a workshop-based conference on all aspects of advancements in software development – <a href="http://spaconference.org/spa2017/themes.html">technology, processes, people and practice</a> – and we are looking for people to join the organising committee for the 2018 conference.</p>
<h2 id="joining-a-conference-committee-is-fun-and-good-for-developing-skills">Joining a conference committee is fun and good for developing skills</h2>
<p>The advantage of helping shape conferences is that it gives you an opportunity to use and develop skills that you might not be using in your job, for example influencing people to submit sessions, encouraging and supporting people who might not otherwise feel able to speak at a conference. It means you can shape the conference into one that you would like to go to.</p>
<p>It helps you to understand a conference from the other side of the fence, so when you are applying for conferences you have a better idea of what people are looking for. It also looks good on your CV!</p>
<p>Once you’ve had a bit of experience organising one conference you can get involved in others. For example, after being the programme chair for SPA from 2013-2015 I went on to be on the programme committee for <a href="https://conferences.oreilly.com/velocity/vl-eu">Velocity</a>, <a href="https://conferences.oreilly.com/software-architecture/sa-eu">Software Architecture</a> and <a href="http://continuouslifecycle.london/">Continuous Lifecycle</a>, all of which gave me an opportunity to shape high-profile conferences (and network with with some people I admire!). You also tend to get free tickets to conferences you help organise so it’s another way to get to go.</p>
<h2 id="spa-is-a-unusual-and-interesting-conference-to-be-part-of">SPA is a unusual and interesting conference to be part of</h2>
<p>SPA is unusual in that all the sessions are interactive in some way. There are coding workshops, games to teach you about personal resilience, goldfish bowl discussions about technologies, guided group work sessions on Agile processes and more. For an idea of what the sessions are like, have a look at <a href="http://spaconference.org/spa2017/programme.html">this year’s programme</a>; the keynotes (for example <a href="http://spaconference.org/spa2017/karen-shoop.html">exploring the intersection of knitting and coding, needles provided</a>) or <a href="http://spaconference.org/spa2017/previousconferences.html">previous conferences</a>.</p>
<h2 id="spa-has-a-submission-process-to-encourage-new-presenters">SPA has a submission process to encourage new presenters</h2>
<p>SPA’s <a href="http://spaconference.org/spa2017/submission-stages.html">submission process</a> includes a period where submitters can receive feedback on their submissions before the deadline to allow them to refine their submissions, and all speakers are offered ‘shepherding’ to help make their sessions the best they can be. Submissions are <a href="/jfdi/how-anonymity-affected-gender-balance.html">reviewed anonymously</a> to promote diversity.</p>
<h2 id="being-a-conference-or-programme-chair">Being a conference or programme chair</h2>
<p>The committee consists of two conference chairs and two programme chairs and other roles as needed. Traditionally there are two of each, with one moving on every year so you are never thrown in at the deep end.</p>
<p>We’d also very much like a marketing chair though we’ve not had someone in that post this year.</p>
<p>The conference has been running for more than 20 years so there is a great community and many of the existing team will stay in place, so it’s an opportunity to learn the skills involved in running a conference in a supportive environment.</p>
<p>There is a lot of admin and organisational work involved in putting on an event and we are really lucky to have an excellent events executive, Mandy Bauer, at BCS, who manages that side of things. SPA is a specialist group of the BCS, and we also usually host the event at the BCS offices in London. So as a conference chair you can focus on making the conference the best it can be.</p>
<h2 id="the-time-commitment-of-being-on-the-committee">The time commitment of being on the committee</h2>
<p>For programme chairs, the work is usually from around September to March. Initially you’ll be publicising the <a href="http://www.spaconference.org/spa2017/lead-a-session.html">call for submissions</a>, and contacting people to encourage them to submit. Ideally <a href="jfdi/how-to-get-women-speakers.html">you’d reach out to lots of people to encourage a broad range of speakers</a>. Around January, you will be working with people to help give feedback on the sessions (there are lots of people involved in this so you won’t have to do it all yourself), followed by a review period and then working out the programme.</p>
<p>For conference chairs the work is mostly January to June; getting sponsors, inviting keynote speakers, arranging evening diversions (this year you can <a href="http://spaconference.org/spa2017/diversion-build-a-laser-bot-for-cats.html">build a laser bot for cats!</a>) and then running the actual conference on the days themselves.</p>
<p>In addition, there are other, non-chair roles and the commitment here is as little or as much as you can offer – helping spread the word about the CFP, helping give feedback and review sessions, attending the programme meeting if you can, helping out at the conference itself.</p>
<h2 id="contact-us-if-youd-like-to-know-more">Contact us if you’d like to know more</h2>
<p>If any of this sounds like something you’d like to take part in, or you have more questions, please get in touch with <a href="http://spaconference.org/spa2017/organisers.html">the organisers</a>. We are having a meeting about next year’s conference on July 11th, and if you’d like to come to that please do get in touch.</p>
<p>Whether you are interested or not, it’s worth attending <a href="http://spaconference.org/spa2017/">this year’s conference</a>. This year we have sessions on topics as varied as <a href="http://spaconference.org/spa2017/karen-shoop.html">knitting and coding</a>, <a href="http://spaconference.org/spa2017/sessions/session715.html">blockchain</a> and <a href="http://spaconference.org/spa2017/sessions/session707.html">pairing across skill levels without the drama</a>.</p>
Writing a business case2017-03-01T00:00:00+00:00https://www.annashipman.co.uk/jfdi/writing-a-business-case.html<p>As I become more senior I have more of a need to write business cases. I turned to the excellent Peter Grzeszczak for advice, and this is what he said.</p>
<h2 id="start-with-the-problem-not-the-solution">Start with the problem, not the solution</h2>
<p>Instead of opening with the solution you are proposing, start by defining the problem you are trying to solve. This is really useful to clarify your own thoughts on the matter.</p>
<h2 id="make-it-strategic">Make it strategic</h2>
<p>Make the strategic point – why is this a problem that we want to solve? What is the outcome we are trying to get to and how does that support the organisation’s stated aims? In our case, why is this something government should care about?</p>
<h2 id="think-of-three-or-four-potential-solutions">Think of three or four potential solutions</h2>
<p>The reason you are writing this is probably to make a case for doing something that you already think is a good idea. It’s a good exercise to think about some other ways you could achieve the same outcomes. You might realise there is a better way to approach the problem. If nothing else, it demonstrates to the people the business case is aimed at that you have properly considered other options.</p>
<h2 id="your-first-potential-solution-should-be-do-nothing">Your first potential solution should be ‘do nothing’</h2>
<p>What is the cost of doing nothing? Are there any benefits to taking no action? Again, this is very useful for clarifying your own thoughts, as we tend to have a bias towards taking action.</p>
<h2 id="outline-the-costs-and-benefits-of-each-potential-solution">Outline the costs and benefits of each potential solution</h2>
<p>Often the benefits will be “soft” benefits, i.e. intangible, or things it is hard to quantify. For example, something might be good for recruitment, or it might lead to a better alignment with government policy.</p>
<p>If you can give actual or estimated costs for things that’s even better. For example, how much does a developer on average cost and how long would this piece of work need them for? Tangible benefits are also good.</p>
<h2 id="case-studies-can-be-used-to-support-your-rationale">Case studies can be used to support your rationale</h2>
<p>Case studies can be quantitative or qualitative. The former is where you outline how a similar solution to the one you are proposing saved £X and is therefore evidence for your suggestion that your proposal will save £X. The latter is an example where you can’t offer hard figures, and is where you are recognising that your solution may be a leap of faith but your case study shows why you believe it’s a punt worth making.</p>
<p>The former is better, but the latter can strengthen your case if used judiciously.</p>
<h2 id="list-the-risks-and-potential-mitigations">List the risks and potential mitigations</h2>
<p>Make sure you’ve considered what potential extra costs there could be, and how you would make sure they are addressed.</p>
<h2 id="make-your-recommendation">Make your recommendation</h2>
<p>If you’ve made a good case so far, then it should be obvious at this point what your solution is. You’ve considered other options, including doing nothing, you’ve weighed up the costs, benefits and risks, and this is the case for your desired solution.</p>
<h2 id="include-a-management-case">Include a management case</h2>
<p>If they approve this case, how would you manage it as a project? Do you already have an idea of the group of people you would get together to work on it? If so, include these details.</p>
<p>You should also cover measurement. When you evaluate this piece of work at the end, how will you know it’s been successful?</p>
<p>Make it easy for those reading the business case to see that if approved, you would be able to deliver this successfully.</p>
<h2 id="the-process-should-take-time">The process should take time</h2>
<p>Part of the point of making a business case is sharing the information. It shouldn’t come out of the blue for anyone who will be involved in making the decision or supporting the work.</p>
<p>As a rough rule of thumb, for a business case for a significant piece of work Peter suggested you would probably be looking at around a month: you might spend a week or so on discovery, making sure you’re clear on the problem (maybe having a workshop) and working out who will be the sponsor; once you’ve got the options you’ll probably spend another week or two getting the data to support those options, and then a week or two to write it up.</p>
<p>Be loud and open as you go; you want input from people who have something to add, and you want people to know it’s happening.</p>
<h2 id="seek-feedback-as-you-go">Seek feedback as you go</h2>
<p>Good people to ask for feedback are the people who you will need to support it, for example the people in charge of business operations and the senior sponsor. Talk to people to see how they think the case will be received, and whether the information is presented in the way people will want to see it.</p>
<p>It’s also worth (as with most things) finding a critical friend who is not involved to read through it.</p>
<h2 id="the-five-cases-for-a-government-business-case">The five cases for a government business case</h2>
<p>We were not talking about producing a business case for the Treasury, though that’s a lot of what Peter does. Although that is a much more involved exercise, I found it useful to think of the different cases you are required to make for Treasury cases: strategic, economic, commercial, financial and management. It can be useful to think of your problem in those lights, even if you don’t go into all the detail. <a href="https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020">This guide on assessing Treasury business cases</a> is useful for more information on that.</p>
<h2 id="ultimately-its-about-telling-a-story">Ultimately, it’s about telling a story</h2>
<p>Your business case should be a narrative. Here’s the problem. Here’s a good solution. It might cost us money, but here are the reasons that it’s the right thing to do.</p>
Come to the cross-government Open Source meetup2017-01-25T00:00:00+00:00https://www.annashipman.co.uk/jfdi/cross-government-oss-meetup.html<p>We are organising a series of cross-government Open Source meetups to exchange ideas, talk about code we can reuse or collaborate on and build a community around Open Source.</p>
<p><img src="/img/Share-knowledge.png" alt="Post-it note saying Share knowledge" /></p>
<p>Making code open is the foundation of the transformation of government. One of the major benefits of open code is how easy it makes collaboration, and potentially reuse; saving other teams time and effort. As I wrote about in <a href="https://governmenttechnology.blog.gov.uk/2016/12/15/next-steps-for-open-source-in-government/">my post outlining the next steps for open source in government</a>, the best way to make that happen is to talk to each other.</p>
<p>The first meetup will be co-hosted with the <a href="https://www.gov.uk/government/organisations/ministry-of-justice">Ministry of Justice</a> (MOJ), on the afternoon of Friday 24th February, at MOJ’s office in London.</p>
<p>There will be short talks from <a href="https://www.gchq.gov.uk/">GCHQ</a>, <a href="https://www.gov.uk/government/organisations/home-office">Home Office</a> and GDS about the Open Source work we’re doing. We will then have some discussion sessions, organised on <a href="https://en.wikipedia.org/wiki/Unconference">open space/unconference</a> principles, so attendees can set the agenda for what they would like to cover. Throughout the afternoon there will be plenty of opportunities to talk to colleagues working on the same things in other departments.</p>
<p>This event is only open to people working in government (both civil servants and contractors are welcome). If you would like to attend, please sign up to the <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">cross-government technical architecture mailing list</a> or ask for the sign up details in the <a href="https://ukgovernmentdigital.slack.com/messages/open-code">cross-government Slack</a>.</p>
<p>I hope to see you there to continue the interesting discussions we’ve started.</p>
<p><em>This post originally appeared on the <a href="https://governmenttechnology.blog.gov.uk/2017/01/25/come-to-the-cross-government-open-source-meetup/">Government Technology blog</a>.</em></p>
Next steps for Open Source in government2016-12-15T00:00:00+00:00https://www.annashipman.co.uk/jfdi/next-steps-for-open-source-in-government.html<p><img src="/img/anna_talk.jpeg" /></p>
<p>I was <a href="https://governmenttechnology.blog.gov.uk/2016/11/18/welcome-to-our-new-open-source-lead/">recently appointed Open Source Lead</a> at the Government Digital Service (GDS) with the aim of making more government code open, improving the reusability of the open code we already have, and helping government as a whole be a better member of the <a href="https://en.wikipedia.org/wiki/Open-source_software">Open Source</a> community.</p>
<p>Making code open is <a href="https://www.gov.uk/government/publications/technology-code-of-practice/technology-code-of-practice">vital to the transformation of government</a>. Working openly also <a href="https://governmenttechnology.blog.gov.uk/2016/09/06/celebrating-sharing-and-reusing-the-digital-marketplace/">supports our work with other governments</a> and last week, the UK government reaffirmed its <a href="https://governmenttechnology.blog.gov.uk/2016/12/14/our-commitment-to-better-open-source-practices/">commitment to making source code open by default</a> at the <a href="https://ogpsummit.org/">Open Government Partnership summit</a> in Paris.</p>
<p>By making our code open and reusable we increase collaboration across teams, <a href="https://gds.blog.gov.uk/2016/12/08/now-we-are-5/">helping make departments more joined up</a>, and can work together to reduce duplication of effort and make commonly used code more robust.</p>
<p>A lot of great work has been done across government on this and it’s clear that developers across government are seeing the opportunity to better meet users’ needs through code reuse. We’ve seen that there is demand for more action and more support through our <a href="https://gdstechnology.blog.gov.uk/2016/11/18/holding-our-next-cross-government-stacktech/">cross-government StackTech</a> events and the <a href="https://gdstechnology.blog.gov.uk/2016/03/04/code-sharing-unconference/">code sharing unconference</a> that took place earlier this year.</p>
<p>In this post, I am going to talk about my first priorities as Open Source Lead and let you know how you can get involved.</p>
<h2>Open Sourcing government code</h2>
<p>Over the past five years, a huge amount of government code has been released under <a href="https://opensource.org/licenses">Open Source licences</a>. This has been great for transparency, collaboration and encouraging good practices. Making things open <a href="https://www.gov.uk/design-principles#tenth">makes them better</a>.</p>
<p>However, most of this code is what we call <a href="https://gds.blog.gov.uk/2012/10/12/coding-in-the-open/">coded in the open</a> rather than <a href="https://en.wikipedia.org/wiki/Open-source_software">Open Source Software</a>. The teams don’t guarantee that they will support it or maintain it in the way Open Source Software needs to be, and a lot of it is not set up to be easily reused.</p>
<p>When code would be useful for other teams there are clear advantages to supporting reuse. For the other teams, and for government in general, the advantage is the chance to save time and money. In these cases, it might be worth taking the <a href="https://gdstechnology.blog.gov.uk/2014/12/19/how-we-moved-vcloud-tools-from-coding-in-the-open-to-open-source/">extra steps to make this code Open Source Software</a>.</p>
<p>There are advantages to the originating team as well. Your code will be used and tested in a variety of environments and there is a greater chance of people finding issues and in many cases helping you to fix them. People who use the code often contribute bug fixes back to the original and these may help set direction and contribute features as well.</p>
<p>However, it can be a lot of work to make the code reusable and maintaining it is an extra overhead, so it’s important to focus the effort on the projects which are meeting the greatest user needs.</p>
<p>Teams across government are already doing great work producing reusable code. For example the <a href="https://designnotes.blog.gov.uk/2016/10/14/introducing-gov-uk-frontend-alpha/">GOV.UK frontend alpha</a>, <a href="https://github.com/gchq/Gaffer">GCHQ’s Gaffer</a> and <a href="https://hodigital.blog.gov.uk/2015/12/22/forms-building-for-reuse/">Home Office Forms</a>, to name just a few. Initially, I will be doing user research to understand what code that has already been written by government would be useful more widely, and I will then identify a few projects to focus on making into Open Source Software. There will be opportunities to get involved in this user research which I will talk about more next year.</p>
<h2>Not all code needs to be Open Source Software but all code needs to be open</h2>
<p>Even where the project does not meet user needs for reuse beyond its originating team, it’s worth making it <a href="https://github.com/alphagov/styleguides/blob/master/use-of-READMEs.md">well documented</a>, with <a href="https://github.com/alphagov/styleguides/blob/master/git.md">good commit messages</a>, and blogging and talking about it, so that other teams can reuse your learnings if not the code itself. A great example of this recently is the <a href="https://sfadigital.blog.gov.uk/2016/11/17/when-build-a-thing-really-works/">digital apprenticeship service using GOV.UK’s smart answers code</a>.</p>
<p>There are many benefits to making your source code open even if not fully Open Sourced, including encouraging good practices and making it easy for teams to collaborate. All new code written in government has to be <a href="https://www.gov.uk/service-manual/service-standard/make-all-new-source-code-open">open by default</a>.</p>
<p>However, it’s not always easy for teams who aren’t used to it to make this happen. Firstly, it’s clear that our guidance is not as joined up as it could be so I’m going to be working on clarifying that and filling any gaps, and then I’ll look at how to address any other barriers we find through user research.</p>
<h2>It’s all about community</h2>
<p>The most important thing when sharing code and making code open is to talk to others working on same things, share ideas and learn about code you can reuse or collaborate on.</p>
<p>We held a <a href="https://gdstechnology.blog.gov.uk/2016/03/04/code-sharing-unconference/">cross-government meet-up on code sharing</a> earlier this year and some great ideas came out of that. I will be building on this by organising meetups every few months as part of building a community around this work.</p>
<p>The next cross-government open source meetup will be in February, and GDS is co-hosting with the Ministry of Justice (MOJ). There will be a series of short talks from departments on what they are doing around open source, followed by some <a href="https://en.wikipedia.org/wiki/Unconference">open space/unconference</a> sessions. If you work in government and would like to attend (or speak about what you’re doing about Open Source in your department), <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">sign up to the cross-government technical architects mailing list</a> where I will post details next month. I will also blog more about it closer the time.</p>
<h2>Making higher impact contributions to Open Source Software</h2>
<p>We depend a lot on Open Source Software. For example, you can see from the <a href="https://github.com/alphagov/colophon">GOV.UK Colophon</a> a small fraction of the Open Source Software we use at GDS. Many teams contribute back patches to help improve these projects, but next year I’m going to be looking into how we can make higher impact contributions. This will help make sure that the Open Source Software that the government depends on is more stable in the long term; and also, giving back to these projects that we use for free is the right thing to do.</p>
<p>It’s worth mentioning that the primary focus of my job is not about driving adoption of Open Source Software. Open Source Software is already used widely across government. The <a href="https://www.gov.uk/government/publications/technology-code-of-practice/technology-code-of-practice">Technology Code of Practice</a> is very clear that you must give it equal consideration when choosing technology, and the <a href="https://www.gov.uk/service-manual/agile-delivery/spend-controls-check-if-you-need-approval-to-spend-money-on-a-service">spend controls</a> team are doing an excellent job making sure Open Source Software is given a level playing field.</p>
<h2>How you can get involved</h2>
<p>If you are in government and would like to attend the meetup in February, please sign up to the <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">cross-government architects email (google) group</a> where we will post more details next month. There is also a <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">cross-government slack</a> with channels for a range of topics, including #open-code.</p>
<p>If you are interested in helping us with our user research on any of this <a href="https://docs.google.com/forms/d/14C1OMrcWlaIupekIl9Vrqd1fPCM46HBaLpPUlyBiPKc/edit">please get in touch</a>. I will also be talking next year about the specific work we are doing and how you can take part.</p>
<p>There is lots to do and these are just the first few things I’m focusing on. I’d be very happy to hear your thoughts.</p>
<p><em>This post originally appeared on the <a href="https://governmenttechnology.blog.gov.uk/2016/12/15/next-steps-for-open-source-in-government/">Government Technology Blog</a>.</em></p>
I am now Open Source Lead2016-11-18T00:00:00+00:00https://www.annashipman.co.uk/jfdi/open-source-lead.html<p>I’m pleased to announce that I’m now Open Source Lead at GDS. James Stewart has written <a href="https://governmenttechnology.blog.gov.uk/2016/11/18/welcome-to-our-new-open-source-lead/">a blog post about my appointment</a>.</p>
How to blog2016-09-16T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-to-blog.html<p>Someone recently asked my advice on blogging, particularly how I decide what to write about and whether I set aside specific time. Here’s what I said.</p>
<h2 id="share-what-you-learn">Share what you learn</h2>
<p>This is my guiding principle, for blogging and otherwise. Anything you’ve learned is worth writing up, as someone else will then be able to learn it too. Writing it up also helps make sure you really understand it yourself.</p>
<h2 id="write-about-things-you-are-asked-about">Write about things you are asked about</h2>
<p>If someone asks you for information, or you find yourself sharing the same information with different people, that could be worth writing a post about.</p>
<p>One of my most popular posts was <a href="http://www.annashipman.co.uk/jfdi/conference-speaking.html">one which mainly just links to other resources</a>. I
recently discovered that it had even been linked to in the call for papers for <a href="http://www.webdirections.org/wd15/cfp.html">Web
Directions Australia</a>. It was basically the blog post form of an email I’d sent several times, trying to persuade people to submit to a conference.</p>
<h2 id="write-about-things-you-want-to-remember">Write about things you want to remember</h2>
<p>I don’t have a very good memory, and one of the strategies I use for remembering
things is to write them up. I have often referred back to posts I have written
myself, for example <a href="http://www.annashipman.co.uk/jfdi/some-regex-in-the-form-of-a-picture.html">on regex</a>, <a href="http://www.annashipman.co.uk/jfdi/roof-hacking.html">on setting up a local wifi</a> and <a href="https://gdstechnology.blog.gov.uk/2014/06/04/using-git-to-refactor-vcloud-tools-into-separate-gems/">on Git</a>.</p>
<p>This helps build up a body of work, and also helps you to be less perfectionist
about what you post. And it’s also very useful when you want to refresh your memory!</p>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr"><a href="https://twitter.com/matwall?ref_src=twsrc%5Etfw">@matwall</a> <a href="https://twitter.com/brunns?ref_src=twsrc%5Etfw">@brunns</a> occasionally i google for an answer to a problem and find one written by <a href="https://twitter.com/philandstuff?ref_src=twsrc%5Etfw">@philandstuff</a></p>— Philip Potter (@philandstuff) <a href="https://twitter.com/philandstuff/status/497434096755548160?ref_src=twsrc%5Etfw">August 7, 2014</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="write-about-things-people-ask-you-to-write-about">Write about things people ask you to write about</h2>
<p>Sometimes people suggest you write a post about something you’ve explained to
them or discussed with them, as the excellent <a href="https://twitter.com/actionjack">Martin Jackson</a> did for <a href="http://www.annashipman.co.uk/jfdi/good-pull-requests.html">my post on how to raise good pull requests</a>. I’m really glad he made that
suggestion as I’ve seen that shared several times since (<a href="https://github.com/gds-operations/vcloud-launcher/pull/99#issuecomment-111081975">often in PRs</a>).</p>
<p>If someone suggests you write a blog post about something that’s always a good hint as it means you’ll have at least one reader!</p>
<h2 id="dont-be-perfectionist">Don’t be perfectionist</h2>
<p>Before I started blogging, I worried for quite some time about what I would say, how I would know to write about, or whether it would be interesting. In the end I decided to just go for it (hence the name of my blog).</p>
<p>My early posts are really very boring; they were just details of me getting to grips with Unix. I doubt anyone read them. But it got me started, and it got me into the habit of writing things up and speaking into the void.</p>
<p>The more you write boring posts, the more you think of how things can be interesting. Don’t wait for ideas to be good enough.</p>
<h2 id="you-could-try-being-disciplined-about-a-schedule">You could try being disciplined about a schedule</h2>
<p>Having a set schedule can help. For example, the excellent <a href="https://twitter.com/catehstn">Cate Huston</a> posts every Monday, Wednesday, Friday and Sunday. <a href="http://www.catehuston.com/blog/2014/04/25/5-strategies-for-making-progress-on-side-projects/">She points out</a> that this means you get better with practice, and also, treating the schedule as more important than the content avoids you making the wrong decision about what will be interesting to other people. (That whole post is useful advice on how to maintain side projects.)</p>
<p>The inspiring <a href="https://twitter.com/markhneedham">Mark Needham</a> started <a href="http://www.markhneedham.com/">his blog</a> by committing to write a blog post every day about one thing he’d learned. He didn’t manage it every day, but over seven years he has managed more than one every other day.</p>
<h2 id="write-the-post-when-you-get-the-idea">Write the post when you get the idea</h2>
<p>However, I don’t have a schedule. My aim is just to keep my site roughly up to date, which for me means I start to feel anxious if I’ve not posted in the past three months. I usually try and write posts when the idea strikes me and if I’ve been writing a lot, I might park it to post later.</p>
<p>For example, I wrote the first draft of this in August 2015, but I had just <a href="https://gdstechnology.blog.gov.uk/2015/07/24/a-career-path-for-technologists-at-gds/">posted on the GDS Technology blog</a> and I knew I had <a href="https://gds.blog.gov.uk/2015/09/08/building-a-platform-to-host-digital-services/">two</a> <a href="https://gdstechnology.blog.gov.uk/2015/10/27/looking-at-open-source-paas-technologies/">more</a> GDS blog posts coming soon, so I parked this one.</p>
<p>Writing a draft of it then was very easy, as I’d just had the conversation offering advice so it was all fresh in my mind, and it was easy to redraft it last night as the meat of it was all there.</p>
<h2 id="structuring-your-post">Structuring your post</h2>
<p>How to write is a whole nother post (/book) in itself but three things I bear in mind are:</p>
<ul>
<li>split the post using headings which can be skimmed to tell a story on their own</li>
<li>
<p><a href="http://www.slate.com/blogs/browbeat/2013/10/18/_kill_your_darlings_writing_advice_what_writer_really_said_to_murder_your.html">Kill your darlings</a></p>
</li>
<li>try and keep the introductory paragraph as brief as possible while still telling the reader what they will find herein.</li>
</ul>
<h2 id="youll-get-into-the-habit">You’ll get into the habit</h2>
<p>Once you get into the habit of blogging and figure out your own style, you start to recognise what things can be formed into a blog post as they are happening.</p>
<p>For example, I’ve written up <a href="http://www.annashipman.co.uk/jfdi/code-sharing.html">discussions at conferences</a>, <a href="http://www.annashipman.co.uk/jfdi/how-to-estimate.html">advice people have given me</a>, <a href="http://www.annashipman.co.uk/jfdi/mobile-browsing.html">work I’ve done</a> and <a href="http://www.annashipman.co.uk/jfdi/preparation-begins-for-javascript-talk.html">work I plan to do</a>. It’s not all <a href="http://www.annashipman.co.uk/jfdi/roof-bug-fixing.html">freeform discussion of why roofing should be more like software development</a>.</p>
<h2 id="the-executive-summary">The executive summary</h2>
<p>If I could only give you two pieces of advice, I’d say: share what you learn, and JFDI.</p>
Making my site look better on small screens2016-07-04T00:00:00+00:00https://www.annashipman.co.uk/jfdi/mobile-browsing.html<p>I’ve had this blog for five years, but I’ve only recently started using my phone to browse the internet, at which point I realised it displayed terribly on a small screen. It’s a wonder anyone ever read my posts.</p>
<p><img class="half-width" src="/img/before_responsive.png" alt="Screenshot of site on an iPhone before redesign" /></p>
<p>As a predominantly back-end developer, it wasn’t immediately clear to me what I needed to do to improve matters, so I thought it was worth making a note here once I figured it out.</p>
<h2 id="responsive-design">Responsive design</h2>
<p>You want the site to respond to information about the client accessing it to display in the best way for that client. In this case, I wanted the site to respond to browsers with a smaller screen and display differently, rather than just show everything as per a desktop browser but just much, much smaller.</p>
<p>The <a href="http://www.w3schools.com/cssref/css3_pr_mediaquery.asp">media query</a> allows the site to get the capability of the device.</p>
<h2 id="redesign-required">Redesign required</h2>
<p>The first thing I needed to do was work out how I wanted the site to look on a mobile device, which actually took a bit of thinking about. I realised that the current layout wasn’t going to work well and, as is often the way of these things, probably already wasn’t working well.</p>
<p>I was using a <a href="http://matthewjamestaylor.com/blog/perfect-3-column.htm">three column layout</a>. However, on some pages the right column was left blank, and on one page I was instead using a two column layout. Only one page was making full use of the three columns. It was time to let it go.</p>
<p><img class="half-width" src="/img/only_page_using_3_columns.png" alt="Only page using 3 columns" /> <img class="half-width" src="/img/page_using_2_columns.png" alt="Page using 2 columns" /></p>
<h2 id="redirecting-a-url">Redirecting a URL</h2>
<p>I took that opportunity to also rename the Games page. I used to spend more time developing little games; now I do a more diverse range of side projects so I can showcase more of that here. Because my site is hosted on GitHub pages I could not do a <a href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes">301 redirect</a>, but I set up a <a href="https://en.wikipedia.org/wiki/Meta_refresh">meta refresh tag</a> to redirect to the new page. A 200 to a 200 is not ideal, but is <a href="https://www.w3.org/Provider/Style/URI.html">better than a 404</a>.</p>
<p>You can see the redesign changes I made <a href="https://github.com/annashipman/annashipman.github.io/pull/5">on GitHub</a>.</p>
<h2 id="use-jekyll-for-the-whole-site">Use Jekyll for the whole site</h2>
<p>When I originally started this blog I handcrafted the HTML for each post, and the rest of the site was also handcrafted HTML. My principle was just to get started rather than waiting until I’d figured out the best way to do it.</p>
<p>When I <a href="https://github.com/annashipman/annashipman.github.io/commit/b3452315d7ce7a468cb81b590fb131dec412aafb">started using Jekyll</a>, I only used the Jekyll features for the blog. However the redesign from the inconsistently applied three column layout made it much easier to <a href="https://github.com/annashipman/annashipman.github.io/pull/7">Jekyllise the whole site</a> and allowed me to <a href="https://github.com/annashipman/annashipman.github.io/pull/6">remove a lot of the duplication and handcrafting</a>.</p>
<p>These initial changes actually made the site worse on mobile because there was more padding on the right.</p>
<p><img class="half-width" src="/img/in_progress_responsive.png" alt="Screenshot of site even narrower on iPhone screen" /></p>
<h2 id="set-viewport">Set viewport</h2>
<p>The first change after the redesign was <a href="https://github.com/annashipman/annashipman.github.io/pull/9/commits/b144e8effe4919d18857685e1ff8cccdb9f467c3">to add the viewport meta tag</a> with an initial scale of 1. This sets the zoom level at 1:1 so the page is rendered at the width appropriate to the device width rather than zooming out to fit the whole page onto the screen.</p>
<h2 id="make-embedded-video-responsive">Make embedded video responsive</h2>
<p>After setting the viewport initial scale, most individual posts looked good on mobile. However the JDFI page has all the posts on it, and it looked very wrong. All the content was squished to the left.</p>
<p><img class="half-width" src="/img/unresponsive_images.png" alt="JFDI page with all content squished to left" /></p>
<p>It turns out that the code provided by YouTube and SlideShare to embed videos/slides into your site is not responsive; it has a fixed width. This means that the site renders the text correctly for the size of the device, but when it gets to the fixed width video it then zooms out to allow space for it.</p>
<p><img class="half-width" src="/img/fixed_width_video.png" alt="An embedded video pushing the page size out" /></p>
<p>These <a href="https://www.abeautifulsite.net/how-to-embed-youtubevimeo-videos-responsively">two</a> <a href="https://coolestguidesontheplanet.com/videodrome/youtube/">articles</a> were useful in working out how to fix this. I changed the HTML to not have the fixed sizes and <a href="https://github.com/annashipman/annashipman.github.io/commit/a66e09868d39c8d2ff6101bcbb2326075feeeffe">added some CSS</a>.</p>
<p>It also turned out that on Safari (and hence on the iPhone) long lines of code were not broken, leading to the same effect as the fixed-width video, which I fixed with an <a href="https://github.com/annashipman/annashipman.github.io/commit/bc2e73a10a144dff2806914eeadd78b77e954f2b">addition to the viewport tag</a>.</p>
<h2 id="only-have-two-horizontal-columns-if-device-is-large-enough">Only have two horizontal columns if device is large enough</h2>
<p>Once I’d done all this set up I was at the point I needed to be, where I could change the layout based on the size of the device.</p>
<p>To do this, I looked through the CSS for anything that changes the horizontal structure of any of the elements, e.g. width, float etc, and <a href="https://github.com/annashipman/annashipman.github.io/commit/251b806141669593e2722432dc91f8168df35354">put that inside a @media query block</a>. Initially this was just the two columns but I later <a href="https://github.com/annashipman/annashipman.github.io/commit/f2fccec50a5b24d72b5aaa7b99b2646f2b5ddcee">added the Twitter link</a>.</p>
<h2 id="move-columns-into-preferred-order">Move columns into preferred order</h2>
<p>After all these changes, I then <a href="https://github.com/annashipman/annashipman.github.io/commit/24f94b2589805e697d4f788c00fdcf9ed84a594b">changed the HTML</a> to make the columns appear in the order I would like if they cannot be side by side.</p>
<p>There are many other improvements to make to this site but hopefully if you are reading on mobile it’s much easier. Do let me know!</p>
Useful questions for one-to-ones2016-03-30T00:00:00+00:00https://www.annashipman.co.uk/jfdi/line-management-questions.html<p>When I started line managing people at work, three years ago, I got some great advice from the excellent <a href="https://www.linkedin.com/in/abseward/">Andrew Seward</a>. He structured one-to-ones around the following list of questions:</p>
<div class="pretty-code-sample">
<ol>
<li>How's it going? (they always say 'fine')</li>
<li>How are you feeling about the work the team's been doing?</li>
<li>Of that work, how are you feeling about the parts you've personally been involved with?</li>
<li>Do you have everything you need to do your job? (important one!)</li>
<li>Is there anything you’d like from me that would help you do your job better?</li>
<li>How is ${new person} doing? (if they have a new team-mate)</li>
<li>Have you had a chance to work with the others in your team? How has that gone?</li>
<li>How have you been getting along when working with other teams within development?</li>
<li>How about when you've had to deal with other parts of the company? How did that go?</li>
<li>${offer some feedback – always some positive and some negative if necessary}</li>
<li>Is there anything else you wanted to raise?</li>
</ol>
</div>
<p>When I started out I didn’t try and subtly steer the conversation to cover these areas, I just explained what I was doing and then went through the questions one by one. It became more natural in later one-to-ones, but even just going through the questions in a very structured way sparked discussion and brought out issues that might otherwise not have come up.</p>
<p>All of my reports who have since taken on line management duties have asked me to share this list, which suggests that it was a useful tool for them too. I’d be interested to hear similar questions that have worked in your one-to-ones.</p>
Code sharing in large organisations2016-03-24T00:00:00+00:00https://www.annashipman.co.uk/jfdi/code-sharing.html<p>At <a href="http://www.scalesummit.org/">Scale Summit</a> last week I led a session on code-sharing in large
organisations. I was particularly interested in how other organisations raise
the visibility of shareable code. This was one of the main themes that came out of the recent <a href="https://gdstechnology.blog.gov.uk/2016/03/04/code-sharing-unconference/">code-sharing unconference at
GDS</a>.</p>
<p>These are the main things I took away from the discussion. For more comprehensive notes, check out <a href="https://github.com/bazbremner/scalesummit-2016-notes/blob/master/code_sharing.org">Barrie Bremner’s write-up</a>.</p>
<h2 id="some-ideas-of-how-to-raise-visibility-of-code">Some ideas of how to raise visibility of code</h2>
<p>People in the session shared the way they publicise code/projects in their organisation. Techniques include writing blogs, either internal or external, presenting at internal meetings and writing things up on internal forums. Facebook have a newsletter called the <a href="http://blog.newspaperclub.com/2014/04/22/paper-of-the-month-the-weekly-push/">Weekly Push</a> which is used, among other things, to publicise less well known features or new products. In some offices this is pinned up in the toilets!</p>
<p>However, apart from the last one, these are mostly pull methods of communication. You have to know they are there and how to find them. The newsletter is a great idea, but in less joined-up organisations you can’t be sure that push communications will get to everyone.</p>
<h2 id="its-useful-to-have-a-community">It’s useful to have a community</h2>
<p>The discussion kept returning to the value of having a community around areas of interest. If you are regularly talking to other people in your field, you are more likely to find out about similar work other people are doing. There can be a lot of noise, and a community can help you learn what works and what doesn’t, and can make it easier to talk to relevant people at the start of a piece of work, rather than when you’ve gone too far down a certain path. In order to encourage useful sharing of information and discussions taking place at the right time, an organisation could support these kinds of communities.</p>
<p>As well as meetings and distribution lists, people talked about other ways to share information, for example, having a <a href="https://slack.com/">Slack channel</a> for a particular community. You could drop in and say “I have this problem, does anyone have a solution?”.</p>
<p>One person pointed out that we often focus on code-sharing when in fact the real goal should be community. If code is shared or reused as a result of that community, that is good, but having a community is where the real value lies.</p>
<p>For those working in UK government, there are already <a href="https://gdstechnology.blog.gov.uk/join-the-conversation/">some forums for these discussions</a>.</p>
<h2 id="how-to-build-a-community">How to build a community</h2>
<p>One useful idea came from <a href="https://twitter.com/bigpg">Paul Gillespie</a> (quoted with permission). He explained that at Skyscanner, they copy <a href="http://www.scribd.com/doc/113617905/Scaling-Agile-Spotify">Spotify’s org struture</a>, and have something called guilds. These are essentially communities of interest. For example, they have one for web dev, one for AWS, one for Python. These guilds have Slack channels, mailing lists and every two weeks they have a guild meeting, which is a one-hour webinar. They use Trello to set the agenda for this meeting, and each guild is run by a core committee.</p>
<p>I later talked to Paul a bit more and he said that in total they have around 6 or 7 guilds. You don’t want to have too many, because being involved in a guild is quite labour-intensive.</p>
<h2 id="beware-of-premature-optimisation">Beware of premature optimisation</h2>
<p>Early on in the discussion it was pointed out that we should be mindful of what problem we are actually trying to solve before addressing how to share code. Many problems seem superficially similar but turn out not to be, so the same code/product/solution will not be appropriate to both. You may waste time coming to this conclusion or bloat software by trying to make it cover too many different scenarios.</p>
<p>There can also be a lot of pressure to consolidate, and some of this can come from senior management seeing false patterns, for example “too many case management systems”. It was noted that spotting actual duplication and finding prior art in this area is part of the role of an architect, but in a large organisation visibility of code is still difficult.</p>
<p>The problem we are trying to solve is to reduce duplication of work, rather than specifically duplication of code. More generally, we do not want teams to waste time reinventing the wheel. We do not necessarily want “the best tool for the job”, we want the most cost-effective tool, and that might be copying someone else’s code, or the team solving the same, or similar, problem in a different way.</p>
<h2 id="code-sharing-isnt-always-the-answer">Code-sharing isn’t always the answer</h2>
<p>If someone has written code that is perfect for your use case, it can still be hard to share. Even if it is well documented, there is still a ramp up cost to understanding it, and it is unusual that the code can be dropped right in. Several people mentioned that you need to weigh up the cost of these factors against the cost of duplication of code. Arrayed against these factors, building it yourself might not turn out to be that expensive.</p>
<h2 id="its-important-to-weigh-up-the-costs">It’s important to weigh up the costs</h2>
<p>The general feeling was that forming a community is very useful to prevent duplication of work or code, but it was also pointed out that there are costs. For example the time cost of communication and staying in touch, keeping documentation up to date etc. Again, these may outweigh the costs of duplicating work.</p>
<p>There are other advantages to being involved in communities of interest, but it is worth considering the cost of the time and effort. For example, while the idea of a Slack channel for a community, mentioned above, can be very useful, <a href="http://m.signalvnoise.com/is-group-chat-making-you-sweat-744659addf7d#.3dhurm2vu">Slack can also be a drain on productivity</a>.</p>
<p>We also returned a few times to the topic of sharing services, or platforms, rather than the code itself. Instead of writing code to be shared, build a service or platform that provides functionality. However, the question of cost came up again: building and operating a service is expensive and takes skilled people, as well as maintenance costs.</p>
<h2 id="my-take-home-message">My take-home message</h2>
<p>The main thing I took away from this discussion is that you need to be clear about what problem you’re trying to solve and what the costs of the solutions are. Sometimes an increased awareness of code that has already been written will solve your problem, but sometimes what you need might be access to a service, or it might be to share knowledge across a community.</p>
<p>Thanks very much to all who took part in the discussion.</p>
Choosing Cloud Foundry for the Government Platform as a Service2015-12-17T00:00:00+00:00https://www.annashipman.co.uk/jfdi/choosing-cf-for-government-paas.html<p>We previously wrote about <a href="https://gds.blog.gov.uk/2015/09/08/building-a-platform-to-host-digital-services/">looking into offering a
platform</a> to host digital
services. This post is about the technology we chose for the beta and how we
made that decision.</p>
<p><img src="/img/Government_PaaS_team.jpg" alt="Government PaaS team" /></p>
<h2 id="comparing-technologies-for-the-prototype">Comparing technologies for the prototype</h2>
<p>The first thing we did was look in detail at the various open source and
proprietary options available. I’ve previously written about <a href="https://gdstechnology.blog.gov.uk/2015/10/27/looking-at-open-source-paas-technologies/">how we compared the
open source
options</a>
and I mentioned that the front-runners were <a href="https://www.cloudfoundry.org/">Cloud
Foundry</a>,
<a href="http://deis.io/">Deis</a> and <a href="https://tsuru.io/">Tsuru</a>.</p>
<p>Deis was a very good option, but we ruled it out for our prototype for two
reasons: it didn’t have the granularity of user permissions we needed, and it
didn’t have a mature service broker, which would allow us to connect to external
services, eg Postgres. Both of these things are on their
<a href="http://docs.deis.io/en/latest/roadmap/roadmap/">roadmap</a>, but the timing
wasn’t going to work for us. However, we had a very interesting conversation
with the Deis team and this is definitely a technology to keep an eye on.</p>
<p>With the proprietary options, the method of comparison was slightly different
because the source code isn’t available to look at, and because it’s not usually
as easy to get a sample platform up and running.</p>
<p>There were four proprietary vendors we were particularly interested in:
<a href="https://www.apcera.com/">Apcera</a>,
<a href="https://enterprise.openshift.com/">OpenShift Enterprise</a> (Red Hat), <a href="http://pivotal.io/platform">Pivotal
Cloud Foundry</a> and
<a href="https://docs.stackato.com/">Stackato</a>. Each one of
the vendors answered questions to a similar level of detail as we were able to
learn by <a href="https://gdstechnology.blog.gov.uk/2015/10/27/looking-at-open-source-paas-technologies/">investigating the open source solutions
ourselves</a>. Because of
commercial confidentiality, I can’t share the detail here but it allowed us to
compare the proprietary solutions, and then again with the open source ones.</p>
<p>They all had advantages, but the one that most suited our requirements was
Apcera.</p>
<h2 id="comparing-tsuru-cloud-foundry-and-apcera">Comparing Tsuru, Cloud Foundry and Apcera</h2>
<p>We wanted to get a prototype up and running quickly so we could start showing it
to our users in government to see if we were on the right lines. So we decided
to start with an open source solution because there are no licensing issues.</p>
<p>We built a prototype using Tsuru, because it’s easier to get started with Tsuru
than Cloud Foundry. Then we used that prototype in our user research - we wanted
to make sure we understood which features were most important to our potential
users.</p>
<p>We then built another prototype in Cloud Foundry to compare its features, ease
of use and maintenance requirements to those of Tsuru. Simultaneously, we spent
some time exploring a trial installation of the Apcera platform with our
engineers providing feedback about each of the three different options.</p>
<h2 id="why-we-decided-to-go-for-open-source-rather-than-proprietary">Why we decided to go for open source rather than proprietary</h2>
<p><a href="https://twitter.com/psd">Paul Downey</a>, the product owner on the
<a href="https://gds.blog.gov.uk/2015/09/01/registers-authoritative-lists-you-can-trust/">Registers</a> team, has described the work
we’re doing on <a href="https://gds.blog.gov.uk/2015/10/07/government-as-a-platform-for-the-rest-of-us/">Government as a
Platform</a> as fitting into three categories:
product, platform and standard.
<img src="/img/productplatformstandard.png" alt="Product, platform and standard sketch by
@psd" /></p>
<p class="photo-credit">Licence: Creative Commons Attribution Paul Downey</p>
<p>This is how we apply those categories:</p>
<ul>
<li>product: we would make the code available and you could use it to build your
own PaaS, which you would then run</li>
<li>platform: we would offer our PaaS as a platform for you to use; this would
incur some costs but would also come with some level of support for the platform</li>
<li>standard: we would define standards for you to build your own PaaS</li>
</ul>
<p>If the technology we choose is not open source, the product would be the
proprietary option. So in effect, we’d just be recommending a product to other
departments, and they would then have to build their own platform and incur
costs over and above supporting and developing the platform. But having the code
available for use by other departments as a platform would encourage
collaboration with our colleagues in those departments.</p>
<p>Using an open source solution brings a number of other important benefits. The
chance to contribute code upstream means we can help make the product useful to
us and others in a similar position. Maintaining and building open source
components and the ‘glue’ (ie the code required to keep it all together) builds
skills and learning in-house. And it also echoes our <a href="https://www.gov.uk/design-principles#tenth">tenth design
principle</a>:
‘Make things open: it makes things better’.</p>
<h2 id="choosing-cloud-foundry">Choosing Cloud Foundry</h2>
<p>The three technologies we looked at in more detail had a lot to recommend each
of them.</p>
<p>Tsuru:</p>
<ul>
<li>has a very simple architecture and is easy to set up and maintain</li>
<li>uses commonly understood components like MongoDB and Redis</li>
<li>it’s easy to swap these components for others, eg Dan Carley made a change
to replace the default Hipache router with Vulcand</li>
<li>the team are extremely responsive and got back to us quickly on all issues
and pull requests, eg Dan’s Vulcand change was merged and published in the next
available release</li>
</ul>
<p>Cloud Foundry:</p>
<ul>
<li>has a mature team management model and an authentication engine that meets
our needs</li>
<li>handles scaling easily and effortlessly, performing well under a heavy load</li>
<li>like Tsuru, it has a mature service broker framework, but there are more
services already implemented (e.g. MySQL) for Cloud Foundry, which means less
work for us</li>
<li>has a very large community with plenty of opportunities for improvement,
sharing tools, recruiting specialists, and sharing knowledge</li>
</ul>
<p>Apcera:</p>
<ul>
<li>has robust multi-tenant support governed by a policy-driven model that
allows granular control of resources, including networks and services, packages,
versions and even regions</li>
<li>has fine-grained control, which makes detailed auditing easier</li>
</ul>
<p>While each of the technologies we looked at had advantages, the open source
requirement is important to this project, so we had to rule Apcera out for now.</p>
<p>It was a very close contest between Tsuru and Cloud Foundry. After a lot of
consideration we chose Cloud Foundry for our beta. The maturity of Cloud
Foundry, as well as the size of its community, were the most significant factors
in this decision.</p>
<p>However, all three technologies are very good and if your team’s requirements
are similar to ours then it’s definitely worth considering them all.</p>
<h2 id="next-steps">Next steps</h2>
<p>We hope to share more detail about what we learned about each technology in
future posts, but in the meantime we’re now starting the beta build using Cloud
Foundry. We’re coding in the open, and aiming to be hosting live services early
next year.</p>
<p><em>This post originally appeared on the <a href="https://governmentasaplatform.blog.gov.uk/2015/12/17/choosing-cloudfoundry/">GDS Government as a Platform
Blog</a></em>.</p>
Video of Operations: a developer's guide2015-11-27T00:00:00+00:00https://www.annashipman.co.uk/jfdi/ops-a-devs-guide-video.html<p>Video from my FFconf talk <em>Operations: a developer’s guide</em>.</p>
<div class="embedded">
<iframe src="https://www.youtube-nocookie.com/embed/y6hbrS3DheU" frameborder="0" allowfullscreen=""></iframe>
</div>
<p><a href="/jfdi/slides-from-ops-a-devs-guide.html">Slides and links in previous post</a>.</p>
Slides and links from Operations: a developer's guide2015-11-07T00:00:00+00:00https://www.annashipman.co.uk/jfdi/slides-from-ops-a-devs-guide.html<p>Slides from my talk <em>Operations: a developer’s guide</em>, at <a href="http://2015.ffconf.org/">FFConf
2015</a>.</p>
<p>The last slide is links; these are copied below so you can actually click on
them!</p>
<p><a href="/jfdi/ops-a-devs-guide-video.html">Video is here</a>.</p>
<div class="embedded">
<iframe src="//www.slideshare.net/slideshow/embed_code/key/39M8UxzdHZYV1e" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen="">
</iframe>
<div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/annashipman/operations-a-developers-guide" title="Operations: a developer's guide" target="_blank">Operations: a developer's guide</a> </strong> from <strong><a href="//www.slideshare.net/annashipman" target="_blank">annashipman</a></strong>
</div>
</div>
<h2 id="links">Links</h2>
<h3 id="configuration-management">Configuration management</h3>
<p><a href="https://www.scriptrock.com/articles/the-7-configuration-management-tools-you-need-to-know">https://www.scriptrock.com/articles/the-7-configuration-management-tools-you-need-to-know</a></p>
<p><a href="http://gettingstartedwithchef.com/first-steps-with-chef.html">http://gettingstartedwithchef.com/first-steps-with-chef.html</a></p>
<p><a href="https://docs.vagrantup.com/v2/getting-started/provisioning.html">https://docs.vagrantup.com/v2/getting-started/provisioning.html</a></p>
<h3 id="virtualisation">Virtualisation</h3>
<p><a href="http://searchvirtualdatacentre.techtarget.co.uk/definition/Virtualisation">http://searchvirtualdatacentre.techtarget.co.uk/definition/Virtualisation</a></p>
<p><a href="http://searchservervirtualization.techtarget.com/definition/server-virtualization">http://searchservervirtualization.techtarget.com/definition/server-virtualization</a></p>
<p><a href="http://www.infoworld.com/article/2621446/server-virtualization/server-virtualization-top-10-benefits-of-server-virtualization.html">http://www.infoworld.com/article/2621446/server-virtualization/server-virtualization-top-10-benefits-of-server-virtualization.html</a></p>
<h3 id="using-vagrant">Using Vagrant</h3>
<p><a href="https://www.vagrantup.com/">https://www.vagrantup.com/</a></p>
<p><a href="http://blog.bennycornelissen.nl/otto-a-modern-developers-new-best-friend/">http://blog.bennycornelissen.nl/otto-a-modern-developers-new-best-friend/</a></p>
<p><a href="https://github.com/patrickdlee/vagrant-examples">https://github.com/patrickdlee/vagrant-examples</a>
(Useful getting started examples)</p>
<h3 id="docker">Docker</h3>
<p><a href="http://patg.net/containers,virtualization,docker/2014/06/05/docker-intro/">http://patg.net/containers,virtualization,docker/2014/06/05/docker-intro/</a></p>
<p><a href="https://zeltser.com/security-risks-and-benefits-of-docker-application/">https://zeltser.com/security-risks-and-benefits-of-docker-application/</a></p>
<h3 id="containerisation-vs-virtualisation">Containerisation vs Virtualisation</h3>
<p><a href="http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production">http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production</a></p>
<p><a href="https://www.scriptrock.com/articles/docker-vs-vagrant">https://www.scriptrock.com/articles/docker-vs-vagrant</a></p>
<h3 id="make-instead-of-gruntgulp">Make instead of Grunt/Gulp</h3>
<p><a href="http://blog.keithcirkel.co.uk/why-we-should-stop-using-grunt/">http://blog.keithcirkel.co.uk/why-we-should-stop-using-grunt/</a></p>
<p><a href="https://www.youtube.com/watch?v=0RYETb9YVrk">https://www.youtube.com/watch?v=0RYETb9YVrk</a> (Talk on using NPM as a build tool)</p>
<p><a href="https://blog.jcoglan.com/2014/02/05/building-javascript-projects-with-make/">https://blog.jcoglan.com/2014/02/05/building-javascript-projects-with-make/</a></p>
<h3 id="tools-for-better-dev">Tools for better dev</h3>
<p><a href="http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/">http://www.leancrew.com/all-this/2011/12/more-shell-less-egg/</a> (More detail on the 6-line Unix program)</p>
A PaaS for Government2015-10-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/a-paas-for-government.html<p>I gave a keynote at Velocity EU about the work I’ve been doing on a PaaS for
government.</p>
<div class="embedded">
<iframe src="https://www.youtube-nocookie.com/embed/OLOaq-Xf5zU" frameborder="0" allowfullscreen=""></iframe>
</div>
Looking at open source PaaS technologies2015-10-27T00:00:00+00:00https://www.annashipman.co.uk/jfdi/looking-at-os-paas-technologies.html<p>I’ve been working on a prototype of what a Platform as a Service (PaaS) for
government might look like, as we wrote about in <a href="https://gds.blog.gov.uk/2015/09/08/building-a-platform-to-host-digital-services/">a
previous post</a>. <span id="more-1257"></span> One of the first things we did
was look at the open source PaaS options that were available. This post is about
how we did that and what we learned.</p>
<p><img src="/img/our_whiteboard_comparison.jpg" alt="Comparison table of PaaS" /></p>
<h2>The open source options we considered</h2>
<p>We looked at a range of proprietary and open source options. In this post, I
am focusing on open source. This is because much of the information we learned
about the proprietary options was shared in commercial confidence. I’ll talk
more about the proprietary options we considered and how we compared them in a
later post.</p>
<h2>Exploring the options</h2>
<p>PaaS is a very fast-moving field at the moment and there are a lot of
options. The first thing we did was take some time individually within the team
to identify which options were worth investigating. We based that on previous
experience, things we’d read about, and further research online. We were around
eight people on the team, so we had a lot to draw on.</p>
<p>It’s not always the case that you are comparing like-for-like with PaaS
technologies. For example, <a href="https://www.cloudfoundry.org/">Cloud
Foundry</a> is a fully-featured PaaS, whereas <a href="http://mesos.apache.org/">Mesos</a> is a cluster-management system. While
Mesos on its own has a number of PaaS features, it didn’t meet a combination of
the criteria "developer self-service" and "multi-tenancy" (e.g. no
authentication, access control).</p>
<p>I wanted to investigate Mesos as it’s an interesting technology, so we looked
at ways to combine it with other technologies who offer those features. We chose
combinations of technologies based on what we found to be common combinations.
In this example, you can see we looked at both <a href="http://mesos.apache.org/">Mesos</a> + <a href="http://aurora.apache.org/">Aurora</a>, and <a href="http://mesos.apache.org/">Mesos</a> + <a href="https://mesosphere.github.io/marathon/">Marathon</a> + <a href="https://mesos.github.io/chronos/">Chronos</a> (Mesosphere). </p>
<p>At this stage, we ruled out things that we didn’t think were worth
investigating further (for example, they were nowhere near production-ready) and
worked out some combinations that made sense to look more into. </p>
<p>The longlist of technologies we investigated is:</p>
<ul>
<li><a href="http://deis.io/">Deis 1.3.1</a></li>
<li><a href="http://kubernetes.io/">Kubernetes</a> + <a href="https://coreos.com/">CoreOS</a></li>
<li><a href="https://www.cloudfoundry.org/">CloudFoundry 2.0.0</a></li>
<li><a href="https://www.openshift.com/">OpenShift 2.0</a></li>
<li><a href="https://www.openshift.com/">OpenShift 3.0</a> (this came out during
our evaluation period and was very different to version 2.0)</li>
<li><a href="https://mesosphere.com/">Mesosphere</a> (there was no open source
Mesosphere available but we evaluated it using the component technologies it
uses: <a href="http://mesos.apache.org/">Mesos 0.2.1.1</a>, <a href="https://mesosphere.github.io/marathon/">Marathon 0.8.0</a>, <a href="https://mesos.github.io/chronos/">Chronos 2.3.2</a>)</li>
<li><a href="http://mesos.apache.org/">Mesos</a> + <a href="http://aurora.apache.org/">Aurora</a></li>
<li><a href="http://panamax.io/">Panamax</a> + <a href="https://coreos.com/">CoreOS</a></li>
<li><a href="http://rancher.com/">Rancher</a></li>
<li><a href="https://tsuru.io/">Tsuru</a></li>
</ul>
<h2>Our selection criteria</h2>
<p>In our previous post I outlined the four main criteria we’d identified from
our user research: a PaaS would have to be multi-tenant, self-service, allow
application developer support and be able to run on multiple public clouds. This
had already allowed us to rule out some technologies (for example, a number of
PaaS technologies only run on AWS). We also had some further must-have criteria.
The complete list of our selection criteria is below:</p>
<p><em>Must haves:</em></p>
<ul>
<li>Multi-vendor capabilities</li>
<li>Developer self-service model</li>
<li>Support for scaling application instances with ease (elastic scaling,
manual, self serve)</li>
<li>Support for Linux</li>
<li>Ability to choose application language</li>
<li>Ability to recover from failure of all hosts</li>
<li>Ability to maximise the application availability during underlying host
failure</li>
<li>Zero downtime deploys</li>
<li>Some multi-tenancy capabilities</li>
<li>Access to raw stdout / stderr logs</li>
</ul>
<p><em>Investigation points</em></p>
<ul>
<li>What is involved in deploying applications to this PaaS?</li>
<li>How easy is the maintenance/operation for the team maintaining the
platform?</li>
<li>Is there a hosted option available?</li>
<li>Could the unit of deployment be used without the PaaS?</li>
<li>How well documented is the PaaS? Do they keep their documentation up to
date?</li>
<li>What type of multi-tenancy support is offered?</li>
<li>Is there commercial support/consulting available?</li>
<li>What different levels of access permissions does the PaaS support?</li>
<li>Is it open source?</li>
<li>Does the PaaS provide any database service?</li>
<li>What is the language/tech?</li>
<li>What APIs are available to enable application developers to manage their own
applications?</li>
<li>Is this technology production-ready now?</li>
<li>Is there a cost associated with this and what is it?</li>
<li>How do we get data on which application is using which resources?</li>
<li>Is it possible to back up data from the PaaS itself?</li>
</ul>
<p><a href="https://twitter.com/pbansley">Brett Ansley</a>, our business
analyst, wrote these up very clearly and with acceptance criteria to clarify
what we were looking for. For example, for zero downtime deploys:</p>
<p>Given: an application that is receiving requests<br />
When: a new version of the application is deployed<br />
Then: there is no downtime of the application.</p>
<h2>Comparing against our selection criteria</h2>
<p>We then split into pairs and each pair took a technology in turn to evaluate
it. <a href="https://twitter.com/dancarley">Dan Carley</a>, our tech lead,
outlined some consistent steps to take in each investigation so that we could be
sure each pair was investigating in the same way. For example, to investigate
high availability:</p>
<ul>
<li>High availability (if self-hosted)</li>
<li>Kill one of the hosts (if self-hosted)</li>
<li>Repeat HTTPS requests to application endpoints</li>
<li>Confirm that we have the same number of workers</li>
<li>Restore host</li>
</ul>
<p>Each pair spun up the technology they were using and investigated it. As they
found the answer to each of the selection criteria, they marked it on the
whiteboard (main photograph) so we (and any passers-by) could clearly see how we
were progressing and which technologies had what. If any technology failed a
must-have, the investigation would stop; otherwise it was time-boxed to two
days.</p>
<h2>Multi-tenancy</h2>
<p>The overview of what we learned about each can be seen from the photograph of
the whiteboard above, and is summarised in <a href="https://gdstechnology.blog.gov.uk/wp-content/uploads/sites/31/2015/10/Consolidated-evalution-of-PaaS-Technologies-Sheet1-1.csv">this
spreadsheet</a>. It’s worth noting that the spreadsheet is slightly more
up-to-date than the photograph of the board; for example Rancher and Tsuru were
late entries, and some answers were updated with more information that we
learned later.</p>
<p>One thing that I found particularly interesting was that multi-tenancy is not
a feature of many of these technologies. For example, Kubernetes and Mesos, two
widely used and interesting technologies, do not support multi-tenancy. There’s
no way to ensure that a team of developers can administer only their application
and not the applications of another team. This meant that they were not suitable
for our purposes.</p>
<h2>The tech that meets our needs</h2>
<p>After going through this process of looking at and comparing a number of open
source PaaS solutions, the clear front-runners were Deis, Tsuru, and Cloud
Foundry. The next stage was to investigate these three technologies more and
choose one to build a prototype. This has helped us with user research, which
we'll share more on later. In the meantime, we hope sharing what we’ve learnt
about these technologies is useful to you, and do let us know your thoughts in
the comments below.</p>
<p><em>This post originally appeared on the <a href="https://gdstechnology.blog.gov.uk/2015/10/27/looking-at-open-source-paas-technologies/">GDS Technology
Blog</a></em>.</p>
Building a platform to host digital services2015-09-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/building-a-paas.html<p>Right now, hosting services is one of the most time-consuming barriers for
new digital services, and usually involves duplicating work done elsewhere. On
the Government Platform as a Service team we’re working on solving that.<span id="more-19788"></span></p>
<h2>Repetition, repetition, repetition</h2>
<p>Every digital service that government runs needs to have a place to run from;
it <a href="https://en.wikipedia.org/wiki/Web_hosting_service">needs to be
hosted somewhere</a> so that it is accessible to users via the internet. The
service doesn’t ‘just work’; there is a lot of effort involved in setting up all
the components required to host a service.</p>
<p>These components don’t vary much between services. Every service needs an
automated way to let developers know when something is wrong, or to alert them
to something. So, in practice, these groups of components end up looking very
similar across very different services. The picture below shows you an
example:</p>
<p><img src="/img/Separate-projects-image.jpg" alt="image showing three projects with the same technical stack, including alerting, monitoring, logging, each running on a cloud provider" /></p>
<p>As you can tell, there’s a lot of duplication. Teams all over government can
end up duplicating work that’s already been done elsewhere. That means spending
time on areas that aren’t their speciality, such as application monitoring or
log aggregation, which stops teams from focusing on their areas of
expertise.</p>
<p>It also leads to a lot of time searching for people with expertise in this
area to hire. All of this takes time and money and leaves teams less time to
focus on their users’ needs.</p>
<p>One way to address these issues is to provide a <a href="https://en.wikipedia.org/wiki/Platform_as_a_service">platform as a
service</a> (PaaS) that services could use for their cloud hosting. A shared
PaaS would then change the diagram above into something more like the one
below:</p>
<p><img src="/img/Government-PaaS-Image.jpg" alt="image showing three projects, each running on Government PaaS, which has a technical stack including alerting, monitoring, and logging, and running on three different cloud providers" /></p>
<p>A Government PaaS wouldn’t just solve the issue of duplication, and where to
focus your teams. One thing that takes a lot of time in government is procuring
commercial services and making sure they are accredited. If we could do that
once, for the PaaS, then that could save service teams a great deal of time,
while making sure that those aspects are being handled in the correct way.</p>
<h2>What a Government PaaS needs</h2>
<p>From the user research we’ve been doing it’s clear that it’s important that
<a href="http://whatis.techtarget.com/definition/multi-tenancy">our platform has
a concept of multi-tenancy</a> - applications that run on the platform should be
isolated from each other and not be able to read or change each others’ code,
data or logs. It wouldn’t be appropriate, for example, if the Digital
Marketplace application was able to access the data of the GOV.UK publishing
platform.</p>
<p>We’ve also learned from our experience supporting GOV.UK that a platform
where the <a href="http://www.infoq.com/presentations/gov-uk-devops">people
developing applications also support the application out of hours</a> leads to
better software and a better user experience. We want a platform that supports
this model right from the beginning.</p>
<p>Apart from multi-tenancy and the support model, there are some other things
that we feel are important in a shared PaaS.</p>
<p><strong>It needs to be self-service</strong>. It needs to be easy and quick
for application teams to get started, and the teams using the platform need to
be able to make frequent changes. That means we need to make sure applications
can be deployed and managed by the application teams, but also that they can
make other administrative changes to their applications, for example configuring
DNS. Allowing teams complete control of their applications will remove any
unnecessary delays for them, and means the platform team can focus exclusively
on iterating and improving the platform itself.</p>
<p><strong>It needs to run on multiple public clouds</strong>. This approach
ensures that we avoid being locked into a single provider, so we encourage price
competition, while also removing the risk of a <a href="https://en.wikipedia.org/wiki/Single_point_of_failure">single point of
failure</a>. Changing infrastructure providers is very difficult to do if you’ve
built to a single provider’s specification so this needs to be built in from the
beginning.</p>
<h2>What we've been doing</h2>
<p>We’ve spent a couple of months exploring what a Government PaaS might look
like and how it could help teams running digital services across government.
We’ve spoken to many potential users, and we’ve worked closely with our
colleagues in other departments who are working to address similar problems, and
we’ve found that no existing departmental solution meets all the needs we’ve
identified.</p>
<p>We’ve evaluated several open source and commercial options, and we’ve built a
prototype and shown it to potential users – developers, web operations engineers
and services managers – both within GDS and in other departments. We’ve tested
our prototype by seeing how it works with real applications (for example, we
tested it using <a href="https://www.digitalmarketplace.service.gov.uk/">Digital
Marketplace</a> and GOV.UK’s <a href="https://github.com/alphagov/government-frontend">Government
Frontend</a>).</p>
<p>We’ll write about all of this more in later blog posts.</p>
<h2>What we're doing next</h2>
<p>We expect to be in alpha until the end of November, by which time we will
have completed a detailed comparison of two open source PaaS technologies and
addressed issues around performance, security, and scalability, for example. We
are really interested in talking to more potential users, so if you are
interested in getting involved in our user research, or seeing a demo of what
we’ve done so far, please get in touch.</p>
<p><em>This post originally appeared on the <a href="https://gds.blog.gov.uk/2015/09/08/building-a-platform-to-host-digital-services/">GDS
Blog</a> and was co-written with <a href="https://twitter.com/massacarl">Carl Massa</a></em>.</p>
A career path for technologists at GDS2015-07-24T00:00:00+00:00https://www.annashipman.co.uk/jfdi/technical-career-paths.html<p>Until recently, career development for technologists at GDS was
relatively unstructured, and dependent on what opportunities came up. Over the
last few months I’ve been leading on developing a more structured career path
for developers and web operations engineers. This blog post describes what that
involved.</p>
<h2>Working out our values</h2>
<p>First, we needed to be clear on what skills we expect someone to demonstrate
at each level (e.g. junior, senior). Every organisation has implicit cultural
values, and we wanted to draw these out as well as technical skills, so it’s
clear what behaviours we want to reward at GDS.</p>
<p>It's really important in government that people are well rounded, for example
having the skills to share externally the things we do at GDS, having an ability
to coach and lead colleagues. Making these things explicit is an important part
of the exercise, and is one reason why it’s hard to take a career path from
another organisation – their values might not reflect yours.</p>
<p>To identify these skills and values we held a series of facilitated workshops
with a group of people from our technology community. In order to involve more
of our technologists, <a href="https://twitter.com/rjw1">bob walker</a>, <a href="https://twitter.com/danielroseman">Daniel Roseman</a> and <a href="https://twitter.com/tombaromba">Tom Byers</a> gathered feedback between
the workshops from others in the web operations, backend development and
frontend development communities.</p>
<p>We also decided at this point that there shouldn't be a frontend/backend
distinction; many of the skills are the same, and we want to encourage people to
work across the full stack.</p>
<h2>Adding more technical skills</h2>
<p>The workshops produced an outline of what skills were expected at each level.
However, it was quite light on details of technical expertise. It's harder to
work that out by committee.</p>
<p>Myself, <a href="https://twitter.com/bradwright">Brad Wright</a> and <a href="https://twitter.com/jystewart">James Stewart</a> – three technical
architects who are also senior line managers – listed the technical competencies
we expect at each level. We also talked to line managers to see how what we had
come up with fitted in with how we were already working on developing
people.</p>
<p>At this point we had an alpha version of the career paths document,
definitely not formatted as we would like, but with the information we had
learned in it. You can download the first version as a PDF: <a href="https://gdstechnology.blog.gov.uk/wp-content/uploads/sites/31/2015/07/Developerweb-op-career-paths-v1.0.pdf">Developer:web
op career paths v1.0</a>.</p>
<h2>Use of the career path</h2>
<p>The main aim of the career path work is to help people clarify where they are
in their careers and what to focus on when working out how to progress, so the
way the document is meant to be used is in a conversation with your line manager
to talk about where you want to get to, and what practical things to focus on to
help you get there.</p>
<p>The conversation focuses on the four skill areas: technical expertise,
delivery, digital evangelism and leadership. So for example, you might be a
developer, with technical skills at a senior developer level, but weaker on
evangelism and leadership. If your goal is to get to senior developer, we would
work out some objectives that would help you develop in the evangelism skill
area (for example, blogging, speaking at events) and leadership (for example
being a tech lead, line management, or owning a non-technical project).</p>
<h2>Too much information!</h2>
<p>Once we had version 1.0 of the document, the next thing to do was use the
tech line managers as guinea pigs. I pitched the work so far and the
conversation template at a meeting of the tech line managers (there were about
18 at the time) and asked them to go through the process with their own line
managers and let us know how it went.</p>
<p>This process of user-testing was very useful and we learned a lot.</p>
<p>For example, one thing I had tried to make very clear was that this is not
meant to be a tick-box exercise. It should not be the case that if you can show
one example of each skill at each level, you automatically progress to the next
level. The examples are meant to be an indication of the sorts of things you can
be expected to do.</p>
<p>However, including so much detail at each level made it very hard to not do
it as a tick-box exercise. Contrary to what we had expected, we needed to
include less detail, to make it clearer that they were examples.</p>
<h2>It was a useful framework for working out specific areas of
development</h2>
<p>One important thing we discovered was that even though the document itself
still needed further development, the conversations using it were some of the
most constructive and useful line management conversations we’d had so far.</p>
<p>I line manage four people, and all four of them came out of our conversation
with useful, practical development objectives that they were able to immediately
start working on in order to develop their careers as a technologist within
GDS.</p>
<h2>It’s still a work in progress</h2>
<p>We’ve made some changes based on the feedback from the tech line managers,
and with input from <a href="https://twitter.com/jamesholloway">James
Holloway</a>, then a writer at GDS, we produced the next version, which you can
also download as a PDF: <a href="https://gdstechnology.blog.gov.uk/wp-content/uploads/sites/31/2015/07/Career_pathing_for_technologists_v2.0.pdf">Career_pathing_for_technologists_v2.0</a>.
We’re now using it, and all technologists at GDS have had an opportunity to work
out their career plan with their line managers. There are still many areas for
improvement, and <a href="https://twitter.com/alicebartlett">Alice Bartlett</a>,
<a href="https://twitter.com/edds">Edd Sowden</a> and others are currently
working on the next iteration.</p>
<p>One interesting point is that we may have gone too far the other way on
removing detail. I suspect the way to balance this will be around how we
communicate the process, rather than reaching a genuinely “perfect” amount of
detail (this is another reason this document may not, as it stands, work for
your organisation).</p>
<p>There are also several areas we didn't touch on and will have to be addressed
in future. For example, this is not currently related to salary. The career
development conversation with your line manager is about where you sit at the
moment, where you would like to get to and what steps to take to get there,
purely from a skills and career development perspective.</p>
<p>In addition, at the moment this is separate to the annual review process,
though the areas for development identified here are really useful for working
out development objectives for the annual review process.</p>
<h2>It’s definitely been worth doing</h2>
<p>Working out our career paths has been a really useful process and we are now
doing it for other roles, such as delivery managers and designers. One important
thing we’ve learned is that while the finished artefact is useful, the process
of how you create that artefact is more important. It has to reflect what you
value as an organisation, and this process forces you to be clear about what
that is.</p>
<p>GDS started in 2011 with a small team of 12 people, and then grew very
rapidly until we numbered 600 in 2014. When your organisation is first getting
established, things like structured career development are rarely the top
priority. Now we are maturing and people are thinking about staying on for some
time, it’s critical to make sure we put these things in place.</p>
<p><em>This post originally appeared on the <a href="https://gdstechnology.blog.gov.uk/2015/07/24/a-career-path-for-technologists-at-gds/">GDS Technology
Blog</a></em>.</p>
Tips on writing a technical CV2015-07-19T00:00:00+00:00https://www.annashipman.co.uk/jfdi/cv-tips.html<p>In the last few years I’ve been more involved in recruitment and seeing a lot of
CVs for developer roles has given me some thoughts on what makes a good
technical CV. Here are some tips to improve yours.</p>
<h2 id="make-it-relevant">Make it relevant!</h2>
<p>What I want to know when looking at a CV is whether you have the skills I’m
looking for, and if not, whether you show the potential to gain them. I want to
know what technologies and practices you know, and what you are doing at the
moment. Ideally I’d like the two to be related – if I’m hiring for a Ruby
developer (we’re not at GDS, <a href="https://gds.blog.gov.uk/jobs/">we want good developers of any
discipline</a>) I want to know if you are going to
hit the ground running.</p>
<p>So when you are describing your current or recent roles, make sure you highlight
the things you are doing that are relevant to the job that you are trying to get.
For example if the job spec asks for experience of leading a team, then make
sure there is evidence for this in the description of what you’ve been working
on. Use your cover letter or <a href="https://gdstechnology.blog.gov.uk/2013/12/24/applying-for-a-job-at-gds/">statement of
suitability</a> to draw attention to these
areas. Make it easy for the hiring manager to see that you tick all the right
boxes.</p>
<p>Together a CV and covering letter should leave no doubt that you know what is
wanted and that you can provide it.</p>
<h2 id="address-gaps-and-concerns">Address gaps and concerns</h2>
<p>When you are reading a CV you notice gaps and short jobs, particularly recently.
The good CVs are ones that address these rather than brushing over it. For
example, one CV I saw, their most recent employment was two years ago. Had they
been unemployed since then? No – they had taken some time out to raise a family.
Another one had only been in their present job for three months. Why were they
leaving so soon? They covered this in their letter – unfortunately the company
was changing direction due to cash-flow problems. That’s fine – some of the best
people get made redundant. But if you don’t explain, it invites the reader to
wonder if you failed your probation, or if you changed your mind about working
there and might change your mind about working here.</p>
<p>View anything like that through the eyes of the recruiter and offer an
explanation, rather than attempting to gloss over it.</p>
<p>It’s also worth addressing any gaps in the required skills. In the example
above, if you can’t provide evidence of team-leading from work, can you give
other evidence, for example from captaining a sports team or running a Brownie
pack?</p>
<h2 id="do-what-they-asked-for">Do what they asked for</h2>
<p>Make sure you supply the requested information. In my case, I was reviewing CVs
for a job as a developer at GDS. <a href="https://gdstechnology.blog.gov.uk/2015/07/14/applying-for-a-job-at-gds-update/">Our process has now
changed</a> but at the time
our job adverts said that in order to apply, you need to send “a CV, a
CV cover letter, and a written statement of suitability explaining how you meet
all of the competencies and specialist skills required”.</p>
<p>A very large number of candidates do not include a written statement of
suitability. This is an unusual requirement, but in the civil service it is <a href="https://gdstechnology.blog.gov.uk/2013/12/24/applying-for-a-job-at-gds/">an
extremely important
one</a>
because if you do not demonstrate evidence of the civil service competencies at
some point in the process, we are <em>not allowed</em> to employ you.</p>
<p>OK, so this may not seem very relevant if you’re not thinking of applying to the
civil service any time soon. But it is. As a developer, you want to demonstrate
that you are able to understand users’ requirements, and a good place to start
is with what they actually ask for.</p>
<h2 id="sell-yourself">Sell yourself</h2>
<p>Essentially that’s what your CV is: a document for selling your skills. So make
sure it does that. Say what you have done. Don’t say “I was asked to…” or “My
team were given the task of…” Instead, make it clear that you actively seek
out and take opportunities, rather than just being handed them.</p>
<p>It can be quite hard, as many of us are naturally quite modest, but it’s good to
make it clear that you are a self-starter. It’s also worth phrasing things to
show what your contribution to a team’s achievements were. I am interested in
what your team achieved, but I am not hiring your team, I’m hiring you, and I
want to know what you did to help the team reach those goals.</p>
<h2 id="a-profile-can-be-useful">A profile can be useful</h2>
<p>One thing that I find quite useful when looking at CVs is a profile at the top.
Just a couple of lines that sum up your current role and what you’re looking for.
It’s not essential, but it gives me an overview of what I’m going to read below
and whether this person is likely to be a good fit for the role.</p>
<h2 id="you-dont-have-to-say-everything">You don’t have to say everything!</h2>
<p>I don’t really need to read four pages listing every job you’ve had since 1999. What
I’m interested in is what you can do now.</p>
<p>Your CV should fit onto two pages. I want the current/most recent job,
maybe the one or two before that if you’ve not been at your current place for
very long, and if you do a lot of short contract work then
maybe a bit more. And then summarise the rest – you just want to show a natural progression.</p>
<p>Similarly, very long lists of technologies are not that useful, particularly if
they include things like SVN, Eclipse and Unix. If you are sending your CV to be
included in a searchable database, for example for a recruitment consultancy,
these will help, but if you are applying for a specific job, it is better to
focus on the skills being asked for.</p>
<p>If you’ve worked on a full-stack project, you will have come into contact with a
lot of technologies, and if you’re a good developer I don’t doubt you can pick
up ones you don’t know quickly. Just concentrate on proving to me that you’re a
good developer.</p>
<h2 id="context-is-useful">Context is useful</h2>
<p>What shape has your experience been? A line summarising what a company’s
business is and roughly how big it is is useful. Job titles in tech are very
fluid and something like “senior developer” doesn’t mean the same thing <a href="http://www.theguardian.com/info/developer-blog/2014/aug/28/what-does-it-mean-to-be-a-senior-developer">from
place to
place</a>.
If you tell me how big the dev team is, then that gives me a bit of context.</p>
<p>And give details. Examples are good. You were a Linux admin; how many servers did
you manage? Even better is examples of particular things you did. Don’t just say
“I improved performance”, say “the site was experiencing extensive load
spikes and I was able to diagnose the cause as X and implement solution Y which
led to a reduction of Z%”.</p>
<h2 id="spend-time-on-the-layout">Spend time on the layout</h2>
<p>Yes, I know, you’re a developer not a designer. But get a designer to look over
it if you can. Grey text, a lot of bold, tiny text – these all make it harder to
read. When recruiting for the civil service we read each CV closely even if it
is hard to do so, but in the private sector if your CV is too hard to read that
might be enough for it to be rejected.</p>
<p>And be aware that it might be printed out.</p>
<h2 id="a-word-on-age">A word on age</h2>
<p>How old you are is irrelevant to how well you can do the job, and since the 2010
Equality Act it is <a href="http://www.acas.org.uk/index.aspx?articleid=1841">illegal for employers to discriminate on grounds of
age</a>.
So there is no need to include that extraneous information in your CV. Don’t
include your birthdate. You don’t even need to include dates of your formal
educational qualifications which would allow people to guess your age. And
adjectives like “young” to describe yourself are also odd – you are inviting
the recruiter to discriminate. Don’t do it.</p>
<h2 id="in-summary-make-it-easy-for-them-to-pick-you">In summary: make it easy for them to pick you!</h2>
<p>It is much more cost-effective to rule people out at the CV stage than at the
interview stage. It will probably take around 20 minutes for one person to look
closely at your CV. But if you are invited in for interview, that’s a lot of
extra investment of time on the part of the company.</p>
<p>Because we have a fair process in the civil service, you can be sure your CV
will be properly reviewed. But the private sector is not bound by such rules. If
your application is not great, it’s a much harder sell for the company to make
the investment in interviewing you.</p>
<p>Essentially, you want to make it easy for whoever is in charge of hiring to
choose your CV, because your CV is so obviously relevant to the role they are
hiring for. This does mean that you have to revise your CV and cover letter for
every job you apply for, but you should be used to that. You’re a good
developer, so you are already prepared to <a href="https://www.gov.uk/design-principles#fourth">do the hard work to make it
simple</a>. Right?</p>
<p><i>This post was edited on 12/12/2018 to <a href="https://github.com/annashipman/annashipman.github.io/commit/49a0fdbe">add that your CV should be two pages</a>.</i></p>
How to raise a good pull request2015-04-21T00:00:00+00:00https://www.annashipman.co.uk/jfdi/good-pull-requests.html<p>On our team we always commit code using <a href="https://help.github.com/articles/creating-a-pull-request/">pull requests</a>, for review by someone who hasn’t worked on that code.</p>
<p>I was recently pairing with the excellent <a href="https://twitter.com/actionjack">Martin Jackson</a>. He had made a change to use <a href="https://github.com/bcoe/librarian-ansible">Librarian-Ansible</a> to manage our dependencies; but <a href="https://github.com/alphagov/tsuru-ansible/pull/8">the pull request</a> was difficult to review because most of the changes were in one commit. I paired with him to help make it easier to review, and he suggested I write up the guidelines I shared.</p>
<h2 id="write-good-commit-messages">Write good commit messages</h2>
<p>As an absolute minimum, you should use good commit messages. The <a href="https://github.com/alphagov/styleguides/blob/master/git.md">GOV.UK styleguide on commit messages</a> is a very good summary of how to do this and why.</p>
<p>Essentially, the diff shows you what has changed, and the commit message should tell you why. Ideally, you should make the explanation of why you made that change as if you were talking to someone who is looking at this code in two years, long after you’ve moved on to another project. What may seem obvious to you now, when you are head down in that code, probably won’t be obvious to you next month, let alone to someone else later on.</p>
<h2 id="do-one-thing-in-a-pull-request">Do one thing in a pull request</h2>
<p>The smaller the PR is, the easier it is to review and so the quicker your change will make it onto master. A good way to keep it small and manageable to focus on just doing one thing in a PR. The story you’re working on might involve several changes, but if you can, it’s worth splitting them into individual pull requests.</p>
<p>A clue as to when a PR might be doing too much comes when you’re writing the headline for the PR. If you find yourself saying “and” or “also” or otherwise trying to squeeze in a number of concepts, your PR might be better as two or more.</p>
<h2 id="make-the-pull-request-tell-a-story">Make the pull request tell a story</h2>
<p>When someone is reviewing the pull request, it should tell a story. Take the reviewer along with you. Each step should make sense in a story of how you introduced the feature.</p>
<p>For example, with the Librarian-Ansible change we rebased <a href="https://github.com/alphagov/tsuru-ansible/commit/7547ac0d35a">this commit</a> into <a href="https://github.com/alphagov/tsuru-ansible/compare/d891857...d14bb2f44">this series of commits</a>. Each of those commits is self-contained and comes with a commit message explaining why that step is taken. Taken together, they tell a step-by-step story of how we introduced Librarian-Ansible.</p>
<p>This allows a reviewer to follow along with your process and make it easier for them to think about what you’ve done and whether the changes you’ve made are the right ones.</p>
<p>For example a reviewer might get to the point where <a href="https://github.com/alphagov/tsuru-ansible/commit/d55e7c94">we configure Librarian-Ansible to use something other than the default directory</a> and wonder whether we should instead have changed our Ansible code to refer to the <code class="language-plaintext highlighter-rouge">librarian-roles</code> directory. Without the separation of steps into a story, it would be difficult to see that step and wonder about the change, so that potential review point is lost.</p>
<h2 id="make-it-a-logical-story">Make it a logical story</h2>
<p>Ordering the commits so they tell a story can be quite hard to begin with, especially if you’re not sure how the piece of work is going to play out. After a while, you will get a feel for the flow of work and you’ll have a better idea of what small chunks to commit. Until then (and even then) <a href="http://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#Changing-Multiple-Commit-Messages">git rebase interactive</a> is your friend.</p>
<p>Apart from making the PR tell a story, it’s worth rebasing to keep connected changes together. For example, instead of adding some commits at the end that say “I forgot to add this file” or “Also making the same change here”, it will be clearer for the reviewer if you rebase, and add those changes to the original commit to keep the narrative. I’ve often reviewed a PR and made a comment like “this change also need to be made in X”, only to find that has been done in a later commit.</p>
<p>The cleaner and more logical the narrative of the commits in the pull request is, the easier it is for the reviewer to retain the whole context in their head and concentrate on the important things to review.</p>
<h2 id="provide-as-much-context-as-possible">Provide as much context as possible</h2>
<p>Imagine everyone else on the team has no idea what you have been working on. Ideally you want the pull request notification to arrive with a full explanation of what this change is and why you’re making it, so that anyone can pick it up and review it.</p>
<p>Link to the story it relates to (you can also use a <a href="https://help.github.com/articles/about-webhooks/">webhook</a> so the story or issue is automatically updated). Point to other PRs or issues that are related. Explain what tests you’ve done, and if relevant, what the reviewer can or should test to confirm your changes.</p>
<p>The more context you can provide, the easier it is to review which makes it more likely to be addressed quickly.</p>
<h2 id="make-sure-the-pull-request-when-merged-to-master-is-non-breaking">Make sure the pull request, when merged to master, is non-breaking</h2>
<p>Ideally each commit should be an atomic, non-breaking change. This is not always possible. However, a pull request is a request to merge your code onto master, so you must make sure that it is complete, in that it is a non-breaking change. It should work, and the tests should still pass once your pull request is merged.</p>
<h2 id="never-commit-directly-to-master-no-matter-how-small-the-change-is">Never commit directly to master, no matter how small the change is</h2>
<p>This is a rule on my team, for two reasons: communication and review.</p>
<p>When you have a number of people working on a codebase, you want to communicate to everyone what changes you are making. Raising a pull request is a broadcast to everyone <a href="https://help.github.com/articles/watching-repositories/">watching that repository</a> so even team members who have not been involved in that piece of work can keep an eye on what’s going on so that changes do not come as a suprise to them. You also do not know what everyone’s experience is, and raising a pull request can sometimes trigger useful input from an unexpected source; committing directly to master would have lost you that opportunity.</p>
<p>As well as making sure the whole team can keep an eye on what changes are happening, raising a pull request also allows the team to maintain a level of quality through code review. Some changes are so tiny that they (probably) don’t need a review, but by making a rule that you never commit directly to master, there’s no chance that something that should not have done will slip through the cracks.</p>
<h2 id="an-example-of-a-good-pull-request">An example of a good pull request</h2>
<p>Take a look at this <a href="https://github.com/alphagov/frontend/pull/784">extremely good pull request</a> raised by the excellent <a href="https://twitter.com/alicebartlett">Alice Bartlett</a>.</p>
<p>Alice did the work first, and then pulled out the changes one-by-one to a new branch to make it clear. While doing the work she refactored some code but in the final PR she has put the refactoring up front, to clear the way for the change she wants to make. This makes it much easier to review because the changes don’t then clutter up later commits. There is also a lot of detail in the overview.</p>
<p>Raising pull requests like this takes time, but it is really worth doing; it makes it clear for your reviewers, the rest of your team, and for the programmers who will be working on this codebase long after you’ve forgotten about the changes you made today.</p>
Staying technical2015-03-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/staying-technical.html<p>A few weeks ago I wrote a post about <a href="/jfdi/what-is-a-technical-architect.html">being a technical architect</a>, and one of my main conclusions was that you need to stay technical. But how?</p>
<p><a href="https://twitter.com/philwills">Phil Wills</a> had a suggestion for me:</p>
<blockquote class="twitter-tweet" data-dnt="true"><p lang="en" dir="ltr"><a href="https://twitter.com/annashipman?ref_src=twsrc%5Etfw">@annashipman</a> one of the most important things I did, a long time after becoming an architect, was to stop going to recurring meetings.</p>— Phil Wills (@philwills) <a href="https://twitter.com/philwills/status/572117516870393856?ref_src=twsrc%5Etfw">March 1, 2015</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>At QCon, I asked him more about this. Did he mean <em>all</em> recurring meetings? What about retrospectives? What about stand-ups? We ended up having a mini <a href="http://en.wikipedia.org/wiki/Open_Space_Technology#Guiding_principles_and_one_law">Open Space</a> of our own, joined after a while by a couple of others, and then Phil proposed a session on ‘how to stay technical as a senior techie’ at last Friday’s <a href="http://www.scalesummit.org/">Scale Summit</a>. Here are some of the thoughts I got from that session.</p>
<h2 id="being-interruptible-is-part-of-the-job">Being interruptible is part of the job</h2>
<p>Whether by “the job” we mean technical architect, manager, or senior developer, there was broad agreement that being interruptible is an important part of it. You are in that position because you have the skills to help others and it might be that interrupting you saves other people a lot of time. Twenty minutes of your time on a problem you’ve seen before could prevent someone else spending two days stuck down a rabbit hole, for example. So you need to learn how to be interrupted: how to minimise the negative affect interruption has on your own work and enjoyment of it.</p>
<h2 id="not-losing-context">Not losing context</h2>
<p>A big problem with being interrupted is the <a href="http://i.imgur.com/3uyRWGJ.jpg">loss of context</a>, and we talked a bit about how to mitigate that. One suggestion that comes up frequently is blocking out time, e.g. core pairing hours. This may not work if you can’t always pick the times of meetings you need to attend but can certainly be useful when you can.</p>
<p>Another brilliant suggestion was around doing <a href="http://c2.com/cgi/wiki?PairProgrammingPingPongPattern">Ping Pong pairing</a>. The reason this helps is that if you do get interrupted, your pair can carry on without you – this is not ideal, but if the interruption is unavoidable it does mean that the work continues and it’s relatively easy for you to reload the context when you return.</p>
<p>Other suggestions included working on smaller pieces of code, for example bugs in your issue tracker, rather than large features; and making sure that the code you write is such that you can <a href="http://qconlondon.com/presentation/microservices-software-fits-your-head">keep it in your head</a>. Finally, working on supporting tools that aren’t on the critical path can be a good way to keep coding without potentially blocking the team’s progress.</p>
<h2 id="staying-current">Staying current</h2>
<p>Staying technical is not just about writing code, and we talked about some other ways of staying current. Some people found doing <a href="https://www.coursera.org/">Coursera</a> courses useful, though this does require a huge time commitment, usually outside of working hours, so won’t be sustainable for many people. Reading code, rather than writing it, was suggested, along with keeping up with what people are working on in the ticketing system.</p>
<p>One person talked about the company organising a regular <a href="http://www.slideshare.net/wouterla/coding-dojo-in-5-minutes">Coding Dojo</a> which seems like a great way to keep everyone current on what is being worked on. Another way to do this is <a href="http://mobprogramming.org/mobprogramming-agile-workshop/">mobbing, or mob programming</a>.</p>
<p>The main way I stay current, which I don’t think was mentioned in the session, is via Twitter and some newsletters like <a href="http://www.devopsweekly.com/">Devops Weekly</a>.</p>
<h2 id="leadership-is-not-just-about-knowing-the-latest-technologies">Leadership is not just about knowing the latest technologies</h2>
<p>We finished up with a really interesting discussion around why we were all so concerned about staying technical, and it turned out that many of us were worried about losing what it was that had made us senior in the first place.</p>
<p>People pointed out that being senior is not necessarily being able to write perfectly in whatever cool new language recent graduates are using, it’s reasoning about problems. You are in a position to use your knowledge of similar problems to cut to the heart of the matter, and you can use your experience to ask the right questions.</p>
<p>Practical suggestions included using <a href="http://changingminds.org/techniques/questioning/socratic_questions.htm">Socratic questioning</a> to draw out problems people might not have thought about yet. “What do you think the solution is?” “Why?” “What would you do about X?”. You also should not be afraid to say when you don’t know, and “let’s work it out together”.</p>
<p>But for me, the take-home message here was best summarised by <a href="https://twitter.com/rrees">Robert Rees</a>, who I quote with permission: “The fear is that you lose authority as you lose expertise, but actually that’s not true: authority comes from having delivered many projects.”</p>
<p>I found the discussion really useful and have some practical tips to take away. It was also nice to know that there are many others facing the same issues as me. Thanks to Phil for proposing it and to all who took part!</p>
What is a technical architect?2015-03-01T00:00:00+00:00https://www.annashipman.co.uk/jfdi/what-is-a-technical-architect.html<p>I have wanted to be a technical architect since I started out in IT, and last September I was delighted to achieve this long-held goal. But very quickly I realised that, while it was clear to me that the role of a technical architect is overseeing the technical aspects of large software projects, I really wasn’t sure what that meant I should be focusing on day-to-day. So I turned to some of the brilliant technical architects I know, both colleagues and elsewhere in the industry, for their advice.</p>
<h2 id="being-a-technical-architect">Being a technical architect</h2>
<p>In my first job as a developer, several years ago, I sat down with an architect I really respected, and asked his advice on how to get there. The diagram he drew for me in that conversation is the career plan I’ve been working off since then:</p>
<p><img src="/img/career_plan.jpg" /></p>
<p>But now I am here it turns out there is a lot of variance. Different technical architects do completely different things. Some are hands-on with a project or small number of projects, writing code. Some have a consulting role. Some are inward-facing to the organisation, some outward-facing.</p>
<p>I know that I definitely want to stay technical, so I asked specifically about that and also for more detail of what the role involves. I focused on skills over and above my previous role as senior developer; for example, problem-solving, analytical skills and being a technical authority are all part of the developer role as well. I haven’t included people’s names as these conversations all took place in a series of private chats and emails.</p>
<h2 id="a-tech-arch-does-what-needs-to-be-done">A tech arch does what needs to be done</h2>
<p>One thing that came up from almost all of the people I spoke to was that the technical architect does what needs to be done to move things forward. The role was variously described as “being a grown-up”, “the project shaman”, and <a href="http://www.slideshare.net/reed2001/culture-1798664">“the rare responsible person”</a>.</p>
<p>Of course, that could also be the description of the role of delivery manager and product owner and also, in some ways, any other member of the team. So one important thing I’ll have to think about is how to do what needs to be done to get the thing to work, without taking on too much of the non-technical aspects of that.</p>
<h2 id="you-need-to-stay-technical">You need to stay technical</h2>
<p>I definitely want to stay technical, and in any case some of the people I spoke to felt this wasn’t optional – you have to stay technical to do a good job. “You need to be coding as an architect. Try and spend as much time as you can on writing code.” This person also said “when things are going right, a tech lead and a tech arch are the same thing” and recommended <a href="https://www.thekua.com/atwork/category/tech-leadership/">Pat Kua’s series on technical leadership</a>.</p>
<p>Staying technical while also doing whatever needs to be done to get the thing delivered is going to be tricky. One person advised me “the management stuff can eat up all the time if you let it, so make the time to step back and plan or engineer situations that will force you to do some ‘maker’ stuff.” One suggestion was that in a situation where I am overseeing a number of projects, I could go and visit one of them and then spend the rest of the day coding in one of their meeting rooms.</p>
<p>Someone also suggested that it’s good, if you can, to avoid regular, fixed commitments: “if you have a lot of steady state commitments, your ability to spend time on the most pressing issues tends to be squeezed.”</p>
<p>More specific than staying technical is actually trying to make sure you take the time to work alongside the people actually doing the work. One person who had done some job-hopping from technical architect to developer and back said that one thing the job changes had reminded them of is “how frustrating it can be working at the developer level with decisions being made ‘elsewhere’ in ‘smoke-filled rooms’. Working at that level a little may help you to keep in touch with these sort of issues”. Though you have to take care that people don’t treat you with kid gloves.</p>
<h2 id="responsible-for-communicating-and-enabling-the-technical-vision">Responsible for communicating and enabling the technical vision</h2>
<p>There were a lot of useful general comments on what the core skills of a technical architect are. It’s about owning the overall vision, <a href="https://www.thekua.com/atwork/2014/11/the-definition-of-a-tech-lead/">establishing a common understanding of what the team is aiming for</a>, and communicating it.</p>
<p>The technical architect should be paying attention to whether we are building the right thing, the right way, and the long-term vision. This might involve requirements, and it might involve a roadmap, but ultimately, the architect – as one person put it – should be “fundamentally lazy”; an enabler. Not command and control; rather communicate the long-term vision and let the team figure it out.</p>
<p>And the most important skill of a technical architect is the ability to find simplicity in complexity.</p>
<h2 id="influencing-people">Influencing people</h2>
<p>Many people point out that a technical architect’s role includes influencing people. As in a tech lead role this may well include stakeholder management, persuasion, and and maybe even convincing people to make big changes in the way they work.</p>
<p>One person had some very practical advice on this: “A conclusion from my initial architect days was that influencing people and coaching them is a major part of the job so I needed to work on it. The former doesn’t come naturally so I still have to make a conscious effort to take opportunities to talk to people without a specific purpose in order to form a relationship that will help us work together better later. People are far more of a challenge than computers! As someone said recently, if you’re looking for a complex adaptive system to work with, try people!”</p>
<h2 id="say-no-to-things-you-really-want-to-do">Say no to things you really want to do</h2>
<p>One skill that I’ve had to work hard to develop is learning to say no. Initially, you progress in your career by volunteering for a lot of things. You gain a breadth of experience, leadership skills and get to work on a lot of varied, interesting stuff. However, as people start to recognise you as someone who can get things done, you will be asked to take on more, and you need to learn to be selective in what you agree to take on so that you can properly focus your attention on the things you have taken on.</p>
<p>I thought I was quite good at that, but in my new role, I am going to have to take it to another level. I am going to have to start saying no to things that I really want to do. One person put it like this: “You will get lots of opportunities to do things that you would be really good at, that you would enjoy and you know that you would knock it out of the park; and you need to be able to say no to those things that don’t meet your goals.”</p>
<h2 id="as-a-technical-architect-you-are-more-in-charge-of-your-own-time">As a technical architect you are more in charge of your own time</h2>
<p>Someone said “as a tech arch, you are largely in charge of your own time, so decide what you want to do and do it”. As noted above, a good thing to do is to carve out time for coding, design, or just thinking about a problem.</p>
<p>That doesn’t seem to reflect my current situation though. While it’s true I’m in charge of my time to the extent of how I help the team reach our goals, there seem to be a number of non-optional things I have to do in order to get there. A lot of that involves making sure to talk to people, and my days at the moment tend to be back-to-back meetings. Here is tomorrow:</p>
<p><img class="narrow-image" src="/img/calendar.png" alt="calendar" /></p>
<p>And most of last week was the same.</p>
<p>In part, this is a function of the stage of my project is at; we’re just starting out so still need to communicate our initial goals and build the team. But I think it also indicates that there are some skills that I need to develop as a technical architect.</p>
<p>Luckily I have a great team and I’m confident that we can work out how to give everyone the room to do what they need to do, so my immediate next step will be working with the team on carving out the time for me to stay involved in the technical aspects of the project. I shall report back.</p>
How we moved vCloud Tools from Coding in the Open to Open Source2014-12-19T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-we-moved-vcloud-tools-from-coding-in-the-open-to-open-source.html<p>At GDS, most of our code is publicly available in our <a
href="https://github.com/alphagov/">alphagov organisation on GitHub</a>. We call
this “Coding in the Open” rather than “<a
href="http://en.wikipedia.org/wiki/Open-source_software">Open Source</a>”. As <a
href="//gds.blog.gov.uk/2012/10/12/coding-in-the-open/">James explained in a
previous blog post</a> this is because for most of our software, we are not in a
position to offer the support that true Open Source requires.</p>
<p>However, we do also have some Open Source Software, in our <a
href="https://github.com/gds-operations/">GDS Operations organisation</a> on
GitHub. When we <a
href="https://gdstechnology.blog.gov.uk/2014/05/07/building-tools-to-provision-our-machines/">started
building vCloud Tools</a>, they were “Coded in the Open”, and we wanted to make
them into Open Source Software. Here are the steps we took to do that.</p>
<h2>Ensuring the project has a long term home</h2>
<p>For the first several months of vCloud Tools’ development, the four of us
working on vCloud Tools were a separate team with a separate backlog. However,
once the tools were <a
href="https://gdstechnology.blog.gov.uk/2014/05/21/using-vcloud-tools-to-provision-a-new-platform/">ready
to be used</a>, we wanted to bring them back into the GOV.UK Infrastructure
Team. <a href="https://twitter.com/mikepea">Mike Pountney</a> and I rejoined the
Infrastructure Team with the vCloud Tools backlog, and our colleagues who hadn’t
been working on vCloud Tools started picking <a
href="https://www.gov.uk/service-manual/agile/writing-user-stories.html">stories</a>
up.</p>
<p>Initially this was mostly <a
href="http://en.wikipedia.org/wiki/Pair_programming">pair programming</a> in
order to share the knowledge. A couple of people who were particularly
interested took more ownership, but everyone in the Infrastructure Team
contributed, including the Head of Infrastructure, <a
href="https://twitter.com/massacarl">Carl Massa</a>, who <a
href="https://github.com/gds-operations/vcloud-tools/pull/188">merged the change
that published vCloud Tools to RubyGems</a>.</p>
<h2>Releasing each tool as a Ruby gem</h2>
<p>Packaging the code as a <a href="http://rubygems.org/">Ruby gem</a> makes it
easier for others to get started with using each tool, using a commonly
installed packaging tool. It also ensures that the code follows a relatively
consistent structure. Most people used to working with tools developed in Ruby
will quickly know how to install it, how to use it, and where to start if they
want to make modifications.</p>
<p>I explained as part of <a
href="https://gdstechnology.blog.gov.uk/2014/06/04/using-git-to-refactor-vcloud-tools-into-separate-gems/">a
previous blog post</a> the steps we took to release each tool as a gem.</p>
<h2>Learning from others’ experience</h2>
<p>We already had several Open Source projects, mostly <a
href="https://forge.puppetlabs.com/gdsoperations">Puppet modules</a> and our own
<a href="https://gds-operations.github.io/guidelines/">Open Source
guidelines</a>, but we were interested in learning from others who knew about
maintaining larger projects. One of the main things I learned while <a
href="http://www.annashipman.co.uk/jfdi/running-oss-projects.html">talking to
people with experience in this area</a> was that when running an Open Source
project, you want to optimise for bringing people into your community.</p>
<p>Because of that, we made some changes, one of which was to add contributing
guidelines with the specific aim of being helpful to new contributors. We wanted
to make it easy for people who were unfamiliar with what would be involved, so
we included explicit detail about how to raise a pull request and what we are
looking for. You can see from the <a
href="https://github.com/gds-operations/vcloud-core/blob/master/CONTRIBUTING.md">contributing
guidelines on vCloud Core</a> that we say pull requests must contain tests, but
we offer to help users write them.</p>
<p>To make it easier for contributors and reviewers to see the quality of
submissions, we also added <a href="https://travis-ci.org/">Travis</a>
configuration so that lint and unit tests and are run against all pull
requests.</p>
<h2>Guidelines for communication with contributors</h2>
<p>At the same time we also set some ground rules amongst ourselves, for example
how quickly we should make sure we respond to pull requests. We drew up some
guidelines for reviewing external pull requests. These were on our internal
operations manual, but I have now added these to our <a
href="https://github.com/alphagov/styleguides/blob/master/pull-requests.md">public
styleguide for reviewing PRs</a>.</p>
<h2>Putting vCloud Tools with our other Open Source projects</h2>
<p>As I mentioned, we have other Open Source projects in our <a
href="https://github.com/gds-operations/">GDS Operations organisation</a>, and
once vCloud Tools had met all our Open Source guidelines, we wanted to move the
tools to this organisation.</p>
<p>Since there are six gems and they are related, we discussed moving them to an
organisation called vCloud Tools. This would have some advantages, for example
it would make it easier to add core maintainers who do not work at GDS. However,
after some discussion, we felt that it was important to keep the link to GDS so
that people who wanted to find out what work government was doing would be able
to find their way to vCloud Tools along with other projects. This is also in
keeping with what other organisations who have a lot of Open Source do, for
example <a href="http://etsy.github.io/">Etsy</a> and <a
href="http://netflix.github.io/">Netflix</a>.</p>
<p>We also created a <a href="http://gds-operations.github.io/">website</a> to
showcase our Open Source, with a lot more detail on <a
href="http://gds-operations.github.io/vcloud-tools/">vCloud Tools</a>.</p>
<h2>Moving issues to GitHub</h2>
<p>Another thing that came out of early discussions with other Open Source
practitioners was the importance of public issue tracking. We had features and
bugs in our own issue tracking system, but having them on GitHub issues is
better for users because it means they can find them. For example, if a user
experiences a bug, they can search for it and may find that we’ve fixed it in a
later version or at least that we are aware of it. It also has the advantage
that people might fix our issues for us!</p>
<p>At this stage we also added a <a
href="https://github.com/gds-operations/vcloud-walker/labels/newcomer-friendly">‘newcomer
friendly’ label</a>, for issues that don’t involve an in-depth understanding of
the tools and don’t require access to a vCloud Director environment. This is to
offer people an easier way to get involved in contributing.</p>
<h2>Encouraging early users</h2>
<p>Because we had been coding in the open from the start, we already had some
early users. This was really useful, because we found they picked up things we
had missed. For example, people coming to the project for the first time will
notice onboarding details that you may have forgotten, and can help you clarify
your documentation.</p>
<p>Users of your software also help by finding issues that you might have
overlooked. For example, one of our early users noticed a <a
href="https://github.com/gds-operations/vcloud-edge_gateway/issues/36">discrepancy
between our documentation and our code</a> which turned out to be a bug. This
was really useful because the bug was in a feature that we don’t use on GOV.UK,
so if they had not raised it, we might never have noticed it ourselves.</p>
<p>In order to encourage users, we visited and spoke to teams both inside and
outside government. This early contact helped us make sure we were focusing on
features that would be useful for more than just ourselves, and in some cases
our users even <a
href="https://github.com/gds-operations/vcloud-core/pull/68">added these
features themselves</a>.</p>
<h2>Sharing our experience with others</h2>
<p>We’ve worked closely with some of our colleagues in government, and currently
teams at the <a
href="https://www.gov.uk/government/organisations/ministry-of-justice">Ministry
of Justice</a>, <a
href="https://www.gov.uk/government/organisations/hm-revenue-customs">HMRC</a>
and <a
href="https://www.gov.uk/government/organisations/driver-and-vehicle-licensing-agency">DVLA</a>
as well as external companies are using vCloud Tools for some of their
provisioning.</p>
<p>However, visiting individual teams wouldn’t scale for every team that’s
interested, so in order to share more widely we started talking a bit more about
what we were doing with vCloud Tools publicly. We wrote some blog posts,
including a one by <a href="https://twitter.com/mattbostock">Matt Bostock</a>
which shows how you could <a
href="https://gdstechnology.blog.gov.uk/2014/05/21/using-vcloud-tools-to-provision-a-new-platform/">use
vCloud Tools to provision your environment</a>. Matt and <a
href="https://twitter.com/dancarley">Dan Carley</a> gave a talk at the Ministry
of Justice, and I gave <a
href="http://2014.sotr.eu/speakers.html#annashipman">two</a> <a
href="http://velocityconf.com/velocityny2014/public/schedule/detail/35839">talks</a>
at conferences.</p>
<h2>What happens next?</h2>
<p>We still have work to do. One thing we’ve found difficult is when a
contributor raises a pull request for a feature that is not something we need at
the moment. There is work involved in reviewing the code and testing it, and
when the feature is not a priority for GOV.UK we’ve found that it’s hard to make
a case for doing that work ahead of other things we need to do.</p>
<p>We are really keen to support our Open Source projects, and including
contributors’ code even when it’s not something we will use immediately helps
the tools be more robust and useful. We’ve discussed some ideas around this, and
our current plan is to allot some time every week specifically for working on
Open Source, so that those kinds of features will be a priority during that
time. We will see how we get on and report back.</p>
<p>We’re also aware that a wider range of contributors leads to other
considerations and what that means about things like how we license code is
another post in itself. Watch this space, and as always, your comments are very
welcome.</p>
<p><em>This post originally appeared on the <a
href="https://gdstechnology.blog.gov.uk/2014/12/19/how-we-moved-vcloud-tools-from-coding-in-the-open-to-open-source/"
target="_blank">GDS Technology Blog</a></em>.</p>
How do I get more women to speak at my conference?2014-12-08T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-to-get-women-speakers.html<p>Last year I wrote about <a href="/jfdi/how-anonymity-affected-gender-balance.html">how anonymous submissions affected gender balance</a> at <a href="http://www.spaconference.org/">SPA conference</a>. Short answer: not hugely, but it did have some other positive effects. This year, however, we did have more women speaking at the conference than previously. Here are some suggestions for how to replicate this at your conference.</p>
<h2 id="the-stats-from-spa">The stats from SPA</h2>
<div class="pretty-code-sample">
<h3>2014</h3>
<p>8/39 = 21% of the session leaders were women.</p>
<p>2/14 = 14% of the solo session leaders were women.</p>
<p>3/8 = 38% of the people running more than one session were women.</p>
</div>
<p>If you compare this to the previous <a href="/jfdi/how-anonymity-affected-gender-balance.html">three years of SPA stats</a>, this is both more women speakers and a higher proportion of the total speakers. In addition, unlike the three previous conferences I’d looked at, some women ran more than one session at the conference. In fact, three women did this.</p>
<p>I recently received an email from a friend organising a conference:</p>
<div class="quote">
<p>Hi Anna,</p>
<p>I'm looking for some help in getting more women speaking and participating at ${CONFERENCE}.</p>
<p>I've (again) promoted the Call for Speakers using (most of) the resources mentioned on the CallbackWomen site. I'm also promoting it through local women in technology groups.</p>
<p>I'm also using an anonymised review process and am (as always) being encouraging to first-time speakers.</p>
<p>Thoughts?</p>
</div>
<p>These are all really good things to do, and I recommend any conference organiser does the same, but I did have one more suggestion, and that is outreach.</p>
<h2 id="the-best-speakers-may-not-realise-it">The best speakers may not realise it</h2>
<p>As anyone who has organised a conference knows, there is little correlation between those who are most keen to submit and those who run the best sessions.</p>
<p>When we announced the call for proposals, as well as tweeting it and sending it to various mailing lists, I and my co-chair set about writing individually to people we thought would propose good sessions.</p>
<p>One name that was suggested to me was that of <a href="https://twitter.com/sandimetz">Sandi Metz</a>. At the time, I wasn’t aware that she was the author of the extremely good <a href="http://www.informit.com/store/practical-object-oriented-design-in-ruby-an-agile-primer-9780321721334">Practical Object-Oriented Design in Ruby</a>, but I looked at <a href="http://www.sandimetz.com/">her website</a> and the courses she was running sounded excellent, so I emailed her in the hope that she might be available.</p>
<p>She got back to me, but she was uncertain about submitting to SPA. Not because, as you might think, it wasn’t a big enough deal conference for her, but because she was uncertain that her material was appropriate for the conference. I quote from her original email (with her permission):</p>
<div class="quote">"I hear that SpaConf is a place for leading edge, new thought, experimental stuff, and I feel like I'm doing simple, straightforward, intuitive, well-known, obvious OOP".</div>
<p>I wrote her an impassioned email back explaining why I thought that topic would be brilliant at SPA (for example, it’s only obvious when you know how…) and she decided to submit.</p>
<p>Her <a href="http://www.spaconference.org/spa2014/sessions/session589.html">session</a> got top marks from all reviewers, was one of the best attended, and when the feedback came in, was rated the highest. I went to the session and it was really excellent. And amazingly, Sandi thanked me for persuading her to attend. Later, she said “I would <em>never</em> have submitted to SPA; you made this happen.”</p>
<p>You might think someone as successful and awesome as Sandi would realise that her session would be amazing. But that wasn’t the case. She explained, “I imagine that things that seem obvious to me are already known by everyone else”. And there are plenty of other excellent people out there for whom that is also true.</p>
<h2 id="you-have-to-encourage-women-to-submit">You have to encourage women to submit</h2>
<p>Women are socialised not to put themselves forward (<a href="http://pwq.sagepub.com/content/early/2013/12/20/0361684313515840.abstract">here is a fascinating study</a> on one way to address this), whereas men are more likely to put themselves forward regardless of competence (Clay Shirky describes this useful skill as being an <a href="http://www.shirky.com/weblog/2010/01/a-rant-about-women/">“arrogant, self-aggrandising jerk”</a>).</p>
<p>This won’t translate to no women submitting; many do. But it will be worth your while approaching the women you would like to submit. And not just women but other under-represented groups. People of colour, differently-abled people, really shy people, for example. People who may have bought the hype that your conference is only for the cool newness, or who lack confidence, or who aren’t going to put themselves forward for whatever reason. Don’t optimise your process for people who think they are amazing.</p>
<p>This is borne out by my own experience. <a href="https://twitter.com/ivanrmoore">Ivan Moore</a> asked me seven or eight times over a series of months “when are you going to submit to SPA?” before I finally submitted my <a href="http://www.spaconference.org/spa2012/sessions/session412.html">Data Visualisations in JavaScript</a> session. It was really successful, and I have since done <a href="/cv.html">several other conference talks</a>.</p>
<p>Don’t just look for people who are already on the conference circuit. Ask women you know, people you work with, etc. Everyone has an interesting story to tell about what they are working on, or are researching, or their hobby, or what they are expert in.</p>
<p>You may have to put more work into encouraging those people to submit. It was not an insignificant amount of effort on my behalf to get Sandi to submit, but it was so worth it.</p>
<h2 id="you-need-anonymity-in-selection-to-back-this-up">You need anonymity in selection to back this up</h2>
<p>It’s important to note that if you are reaching out to people to ask them to submit, rather than directly to invite them to speak, you need anonymity to back this up. To quote from the <a href="http://2012.jsconf.eu/2012/09/17/beating-the-odds-how-we-got-25-percent-women-speakers.html">article that inspired me</a>:</p>
<div class="quote">
If you go around encouraging people that are usually under-represented at your event to submit to the CFP and promise an unbiased selection, it ensures they don’t feel that they’ve been picked because of a particular personal feature, rather than the content of their proposals.
</div>
<p>Our process at SPA remained the same in 2014 as it was in 2013. Read <a href="/jfdi/how-anonymity-affected-gender-balance.html">my original post</a> for full details, but submissions remain anonymous until the point a draft programme is produced.</p>
<h2 id="a-not-exhaustive-note-on-how-to-ask">A not-exhaustive note on how to ask</h2>
<p>I like your work on X. Y is relevant to our attendees. I saw your talk on Z at ${MEETUP} and think something similar/the same talk would be brilliant at our conference.</p>
<p>Not “our conference needs more women/people of colour/LGBT speakers”. I am already going to imagine that’s the only reason you asked me. It would be nice to know that it’s my work you’re interested in.</p>
<p>A shout out to <a href="https://twitter.com/fymd">Andy Allan</a>, who organised <a href="http://sotr.eu/">Scotch on the Rocks</a>. A slightly different scenario as he invited me to speak rather than asking me to submit, but it was a flawless handling:</p>
<ul>
<li>he gave me plenty of notice – initial contact was in October for a conference in June</li>
<li>he talked about some of my work in detail to explain why they were interested in having me</li>
<li>the organisers made a big effort with the rest of the programme as you can see from the <a href="http://2014.sotr.eu/#nav-speakers">final line-up</a>, which is actually missing two women who had to drop out very close to the event and were not replaced</li>
<li>quite close to the conference, Andy asked me if I was also interested in being on a panel. I said that I didn’t want to be the only (read: token) woman, and he said that was fine, he’d already also lined up the excellent <a href="https://twitter.com/kitt">Kitt Hodsden</a>. The <a href="https://www.youtube.com/watch?v=4ai4y-Iecfw&feature=youtu.be">final panel</a> was three men and two women</li>
</ul>
<p>If you are trying to encourage people new to speaking, <a href="/jfdi/conference-speaking.html">these resources</a> might help.</p>
<h2 id="in-summary">In summary</h2>
<p>Seek out women and other under-represented groups who do great work in our field. They will not be hard to find. Pick out some examples of their work that show they’d be good for your conference. Contact them and ask them to submit. Be prepared to put some work into encouraging people.</p>
<p>This does take a bit more time and effort than just tweeting your CFP and leaving it at that. But if you want to put on a great conference – and why else are you bothering? – it is worth putting the effort in to seek out the people who have something interesting, new and exciting to say.</p>
Build Quality In2014-09-03T00:00:00+00:00https://www.annashipman.co.uk/jfdi/build-quality-in.html<p>I am a published author! I have written a chapter on DevOps at GDS for the
book <a href="http://buildqualityin.com/" target="_blank">Build Quality
In</a>. 70% of the royalties go to <a href="https://www.codeclub.org.uk/"
target="_blank">Code Club</a> so do think about <a
href="https://leanpub.com/buildqualityin">buying</a> the book.</p>
Some thoughts on preparing an Ignite talk2014-07-07T00:00:00+00:00https://www.annashipman.co.uk/jfdi/preparing-ignite-talk.html<p>I did my first <a href="http://igniteshow.com/">Ignite</a> talk in November last year, based on <a href="/jfdi/roof-bug-fixing.html">roof bug-fixing</a>, and these are some of the things I found useful while preparing.</p>
<h3 id="launch-right-in">Launch right in</h3>
<p>I always start talks with some kind of anecdote or fact to get the audience
interested, rather than a bio. Why should we care who you are until we know whether you’re worth listening to? So usually I introduce myself around the 2nd or 3rd slide.</p>
<p>But <a href="http://russelldavies.typepad.com/home/">Russell Davies</a>, who was incredibly helpful when I was preparing the talk, suggested I go one further – just keep telling the story. When he pitched it to me he said “it would be a brave move…”, which I took as a challenge I had to accept.</p>
<h3 id="you-will-lose-the-beginning-and-the-end">You will lose the beginning and the end</h3>
<p><a href="https://www.youtube.com/watch?v=rRa1IPkBFbg">Scott Berkun’s Ignite talk on giving Ignite talks</a> is very worth watching if you are considering giving an Ignite talk.</p>
<p>The main practical thing I took from it was that you will probably lose most of your first and last slide. Every time I practiced a run-through, I waited at least a few seconds before starting the first slide.</p>
<p>It was a good job I did. On the night, they had been playing music as people ascended the stage, but for some reason, they started mine when I was already on the stage, when I was about to start speaking. But because I’d practiced not starting immediately, it didn’t throw me, and it didn’t put me behind from the beginning.</p>
<h3 id="dont-be-too-tied-to-your-slides">Don’t be too tied to your slides</h3>
<p>More great advice from Russell: don’t be too tied to your slides. Just talk, and if they coincide then it looks like you’re a genius, and if they don’t it doesn’t matter.</p>
<h3 id="do-real-practice">Do real practice</h3>
<p>Another incredibly useful piece of advice I got from <a href="http://userfirstweb.com/328/successful-ignite-presentations/">Jason Grigsby’s post about his Ignite talk</a>
was to do real practice. This is really important. When you make a mistake in a practice, then carry on – practice recovering. You will learn how to improvise until you get back on track – which is one of the most important skills for a successful Ignite talk.</p>
<h3 id="they-want-you-to-succeed">They want you to succeed</h3>
<p>People are at Ignite talks to have fun. It’s a great crowd, like doing a Best Man’s speech (yes, I’ve done one of those too) – the audience want you to do well. That, or fall off the stage.</p>
<h3 id="its-an-excellent-constraint">It’s an excellent constraint</h3>
<p>The format of Ignite talks is a good constraint to encourage creativity and I really recommend you give it a go. I enjoyed it a lot.</p>
<p>When it was finished I’d made quite a few mistakes and missed out things I’d wanted to say, but on balance, I think it went pretty well. Judge for yourself.</p>
<div class="embedded">
<iframe src="//www.youtube-nocookie.com/embed/Q1qWzz6liK0" frameborder="0" allowfullscreen=""></iframe>
</div>
How I keep alphagov/fog updated...2014-06-16T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-i-keep-alphagov-fog-updated.html<p>I refer back to this useful email from <a href="https://www.twitter.com/mikepea">Mike Pountney</a> pretty frequently, even though it’s easy enough to look up, so thought I’d save it here.</p>
<div class="language-plaintext code-sample highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout master &&
git pull &&
git fetch upstream &&
git merge upstream/master &&
git push
</code></pre></div></div>
Using Git to refactor vCloud Tools into separate gems2014-06-04T00:00:00+00:00https://www.annashipman.co.uk/jfdi/using-git-to-refactor-vcloud-tools.html<p>When I started working on <a
href="https://gdstechnology.blog.gov.uk/2014/05/07/building-tools-to-provision-our-machines/">vCloud
Tools</a>, we had most of our functionality in one repository. There wasn’t a
clear separation of concerns – the code was <a
href="http://en.wikipedia.org/wiki/Coupling_%28computer_programming%29">tightly
coupled</a> – and it also meant that a user who only wanted to use one tool to
do one job had to install the whole thing. So I pulled out each functional area
into an individual <a
href="http://rubylearning.com/blog/2010/12/14/ruby-gems-%E2%80%94-what-why-and-how/">Ruby
gem</a>.</p>
<h2>Rewriting history</h2>
<p>When creating the repository for each new gem, I didn't want to lose all the
commit history; it is very <a
href="http://mislav.uniqpath.com/2014/02/hidden-documentation/">useful
documentation</a>. If I had started each gem from scratch, the commit history
would remain in the vcloud-tools repository, but I prefer not to have to go
somewhere else to see the history of what I'm working on right now.</p>
<p>I also didn't want to create each directory side-by-side and then delete the
unnecessary code, as this would make each repository much larger than it needed
to be (as Git stores all the history) and would essentially be several
duplicated projects with different bits deleted.</p>
<p>What I really wanted was to go back in time and to have started with this
structure from the beginning.</p>
<p>Luckily, Git provides a mechanism to do this: <a
href="https://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html">git
filter-branch</a>.</p>
<h2>Creating the new repository</h2>
<p>To get started, I cloned the existing vcloud-tools repository locally:</p>
<div class="code-sample">git clone --no-hardlinks /path/to/vcloud-tools-repo
newrepo</div>
<p>You need the <span class="code-sample">--no-hardlinks</span> flag because when cloning from a
repo on a local machine, files under <span class="code-sample">.git/objects/</span> are linked to
the original to save space, but I wanted my new repo to be an independent
copy.</p>
<p>I then deleted the remote in newrepo. I didn't want to push my new,
pared-down repo over vcloud-tools.</p>
<div class="code-sample">git remote rm origin</div>
<h2>Deleting irrelevant code</h2>
<p>Having made a new repository, I then pruned away any code that was unrelated
to that tool. So for example when pulling out vCloud Core, I pruned away all
directories that didn't contain vCloud Core code.</p>
<p>For this, I used a tree-filter. This checks out each commit and runs a shell
command against it, in this case <span class="code-sample">rm -rf c</span>, where c is an irrelevant
directory or file.</p>
<div class="code-sample">git filter-branch --tree-filter "rm -rf c" --prune-empty
HEAD</div>
<p>Because it's checking out each commit, it takes some time to do it this way
(though it speeds up, as using <span class="code-sample">--prune-empty</span> removes commits that
are left blank after the shell command does its job, so the total number of
commits decreases as you progress through the task).</p>
<p>This command actually allows you to use any shell command you want, but I
found that deleting things I didn't require one-by-one, while time-consuming,
meant that I picked up some things that had been missed, for example <a
href="https://github.com/alphagov/vcloud-tools/pull/93">files in the wrong
place</a> and <a
href="https://github.com/alphagov/vcloud-tools/pull/94">tidy-up</a> that <a
href="https://github.com/alphagov/vcloud-core/commit/9f0d27a3ab1dea3209f9775707393f1a2cd07">needed
to be done</a>.</p>
<h2>Tidying up</h2>
<p>After each time you run this command and prune away files or directories, you
need to do some cleanup. (I just wrote a little shell script and ran it each
time.)</p>
<p>When you run git filter-branch a <span
class="code-sample">.git/refs/original</span> directory is
created, to allow for a restore. These objects will be retained if you don't
remove them so you need to remove the references:</p>
<div class="code-sample">git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1
git update-ref -d</div>
<p>These are usually cleaned up by Git on a scheduled basis, but because I was
going on to remove other folders, I wanted to expire them immediately, and then
reset to <span class="code-sample">HEAD</span> in case that had changed anything.</p>
<div class="code-sample">git reflog expire --expire=now --all
git reset --hard</div>
<p>Then, I forced garbage collection of all orphaned entities.</p>
<div class="code-sample">git gc --aggressive --prune=now</div>
<p>The final line of my shell script just output the size of the
<span class="code-sample">.git</span> folder so I could see it getting smaller as I pruned away
unneeded code.</p>
<div class="code-sample">du -sh .git</div>
<h2>Important warning!</h2>
<p>You need to be extremely careful when rewriting history. It is very important
not to do this on a public repository unless you have a <a
href="https://help.github.com/articles/remove-sensitive-data">very good
reason</a>, as it makes everyone else’s copy of that repository incorrect. So I
waited until it was finished and I was happy with my local version before
pushing it up to a <a href="https://github.com/alphagov/vcloud-core">new
repository</a>.</p>
<h2>Applying gem structure</h2>
<p>For all tools other than vCloud Core, the first thing I had to do was redo
the <a
href="https://github.com/alphagov/vcloud-launcher/commit/c09090668acfef26df713110d05d0f4850c22163">directory</a>
<a
href="https://github.com/alphagov/vcloud-launcher/commit/65178f291a909cdc517549b11e48a5d3731f97e7">structure</a>.</p>
<p>I also had to move the <a
href="https://github.com/alphagov/vcloud-launcher/commit/8c096a3cc70f6d96680080f4697f45d394e2cbbe">file
that loads the dependencies</a>, and during the pruning process is became clear
that we had a lot of dependencies at the wrong level, or <a
href="https://github.com/alphagov/vcloud-launcher/commit/f6327ba559348ca7309226ac2be83abee4316753">
not required at all</a>. Deleting code is very satisfying!</p>
<p>I then added the <a
href="https://github.com/alphagov/vcloud-core/commit/449e8d94a272c9bda73f069713798691c11f84ea">required
files</a> for a gem, for example a gemspec, a licence. At this point, I also
added a <a
href="https://github.com/alphagov/vcloud-net_launcher/commit/2dc595338e044336b92e36a543718789b1fa67c2">CHANGELOG</a>
to help us move the tools to <a
href="https://github.com/gds-operations/open-source-guidelines">open
source</a>.</p>
<h2>Some interesting things about Git</h2>
<p>I discovered some new things. For example, Git is case-insensitive with
regard to file names.</p>
<div class="code-sample">git mv spec/vcloud/fog/Service_interface_spec.rb
spec/vcloud/fog/service_interface_spec.rb</div>
<p>told me:</p>
<div class="code-sample">fatal: destination exists,
source=spec/vcloud/fog/Service_interface_spec.rb,
destination=spec/vcloud/fog/service_interface_spec.rb</div>
<p>You need to force it with the <span class="code-sample">-f</span> flag.</p>
<p>Also, you can copy commits from another repository, as if you were using
<span class="code-sample">git cherry-pick</span> to copy from a branch in the same repository, by <a
href="http://stackoverflow.com/a/9507417">creating a patch and applying it
locally</a>.</p>
<div class="code-sample">git --git-dir=../some_other_repo/.git \
format-patch -k -1 --stdout | \
git am -3 -k</div>
<h2>Then I published the gem</h2>
<p>To enable our continuous integration, I added a <a
href="https://github.com/alphagov/vcloud-core/blob/master/jenkins.sh">Jenkins
job</a> and a<a href="https://help.github.com/articles/creating-webhooks">
Jenkins hook to GitHub</a> so that a commit to master will trigger a build in
Jenkins.</p>
<p>Once I was happy that everything was present and correct and it was ready to
publish, I added a <a
href="https://github.com/alphagov/gem_publisher">gem_publisher</a> <a
href="https://github.com/alphagov/vcloud-core/commit/3934e43f360e5d19ba7c97ec60f7987acfee2912">rake
task</a>, and then <a
href="https://github.com/alphagov/vcloud-core/commit/a6eaa1f9ee800bed3048074c94770cea9e035934">included
that in Jenkins</a>. This means that when a commit is merged to master, if the
tests pass and the version has changed, the new version is automatically
published to <a href="http://rubygems.org/">RubyGems</a>.</p>
<p>Ta-dah! <a href="http://rubygems.org/gems/vcloud-core">vCloud Core</a>.</p>
<p>Finally, I made a <a
href="https://github.com/alphagov/vcloud-tools/pull/116">pull request on vCloud
Tools</a> to remove it.</p>
<h2>Pulling out all gems</h2>
<p>Over a couple of months I pulled out all the gems and the architecture of
vCloud Tools now looks like this:</p>
<img src="/img/vcloud-tools-architecture.jpg" alt="hand-drawn diagram of vCloud Tools
architecture" />
<p>This approach follows the <a
href="http://www.faqs.org/docs/artu/ch01s06.html">UNIX philosophy</a> of simple
tools that do one thing, which together can form a toolchain to do do more
complex tasks. vCloud Core now takes care of the interactions with fog and the
vCloud API, and the other tools depend on that. <a
href="https://github.com/alphagov/vcloud-tools">vCloud Tools</a> is now a
meta-gem that pulls in all the other gems as dependencies. This has definitely
made it easier to develop on and use vCloud Tools, and I learned a lot about Git
and Ruby gems along the way!</p>
<p><em>This post originally appeared on the <a
href="https://gdstechnology.blog.gov.uk/2014/06/04/using-git-to-refactor-vcloud-tools-into-separate-gems/"
target="_blank">GDS Technology Blog</a></em>.</p>
Sketchnoting2014-05-17T00:00:00+00:00https://www.annashipman.co.uk/jfdi/sketchnoting.html<p>At <a href="http://2014.scottishrubyconference.com/">Scottish Ruby Conference</a> earlier this week, among the many great talks I attended was <a href="http://twitter.com/jessabean">Jess Eldredge</a>’s excellent <a href="https://speakerdeck.com/jessabean/sketchnoting-creative-notes-for-technical-content">introduction to sketchnoting</a>.</p>
<p>I liked the idea of doing sketchnotes since I came across the first ones I saw, by <a href="http://www.yahnyinlondon.com/">Amanda Wright</a>, but had never properly tried it. I do makes notes in a fairly visual way – here, for example, are my notes from <a href="https://twitter.com/kerrizor">Kerri Miller</a>’s excellent talk on concurrency.</p>
<p><img src="/img/concurrency.jpg" alt="concurrency notes" /></p>
<p>The main thing of course is that my handwriting is atrocious.</p>
<p>I was right at the front in Jess’s talk and felt self-conscious, so the notes I made were actually worse than my usual standard.</p>
<p><img src="/img/original_sketchnotes.jpg" alt="original sketchnotes notes" /></p>
<p>However, I was inspired, and also happened to have a nice pen with me, so tried to do a sketchnote in my last talk of the day, <a href="https://twitter.com/indirect">André Arko</a>’s very interesting <a href="https://speakerdeck.com/indirect/development-was-the-easy-part-scottish-ruby-2014">Development was the easy part</a>.</p>
<p><img src="/img/tip_of_the_iceberg.jpg" alt="tip of the iceberg notes" /></p>
<p>Not amazing. But better than my usual notes.</p>
<h2 id="what-i-like">What I like</h2>
<p>Jess’s advice was very practical and could be immediately applied. For example, the title is the bit that looks really good, and that’s because I got there early as she suggested and wrote it before the beginning of the talk.</p>
<p>Her main advice for bad handwriting was to slow down – and I did, and it really helped, in that I can actually read this, unlike my usual scrawl.</p>
<p>I did find it made me concentrate a bit more as everything André said I was trying to figure out what the key point would be to that section so I could write that down.</p>
<h2 id="what-i-didnt-like">What I didn’t like</h2>
<p>However, I clearly ran out of space. Data stores in the bottom right (above request time) was the last section he covered. The bit I missed out was that <a href="http://redis.io/topics/sentinel">redis-sentinel</a> runs an election and while it’s running, all candidates accept writes, but then when one is elected, all the unelected ones throw away their writes.</p>
<p>It’s a shame that this is the one thing I was unable to write clearly, as that was the only thing he said that was completely new to me. That made me realise that I was worrying too much about how the notes would look to other people and whether they’d be able to follow them, so writing things down that I already knew. Whereas one point Jess made several times was these are <em>my</em> notes, and don’t need to make sense to anyone else.</p>
<p>Also, a lot of things I note down in talks are actually items for my To Do list, either unrelated if my attention is wandering or, if I’m engaged, how I can apply these things in my projects. I probably don’t want to capture all of that for posterity.</p>
<h2 id="so-what-did-i-learn">So what did I learn?</h2>
<p>It’s worth taking a nice pen and some plain paper to a conference, because at least I can make notes that will be easier for me to read when I’m looking back at them. I might even buy an artist’s pen, as I really like the shading Jess did.</p>
<p>If I slow down, my handwriting is almost legible!</p>
<p>It’s fun. I always fancied doing the bits of comic-book drawing that didn’t require artistic talent, like cross-hatching – this is my chance!</p>
<h2 id="another-go">Another go</h2>
<p>I decided to write up the notes from Jess’s talk again:</p>
<p><img src="/img/better_sketchnotes.jpg" alt="better sketchnotes notes" /></p>
<p>Better!</p>
Building tools to provision our machines2014-05-07T00:00:00+00:00https://www.annashipman.co.uk/jfdi/building-tools-to-provision-our-machines.html<p>Over the last few months, I’ve been leading a small team <a title="vCloud
Tools" href="https://github.com/alphagov/vcloud-tools">building tools</a> to
automate creation and configuration of our servers.</p>
<h2>Tools to automate provisioning</h2>
<p>Currently, our production environment is hosted with Skyscape, and we manage
it using <a href="http://www.vmware.com/uk/products/vcloud-director">VMware
vCloud Director</a>. Skyscape, and other providers using vCloud Director, expose
a UI for managing VMs and networks.</p>
<p>However, a UI isn't suitable when you need to bring a lot of machines up in a
repeatable, reliable way, for example when <a
href="https://gdstechnology.blog.gov.uk/2014/03/28/migrating-govuk-infrastructure/">migrating
the site to a new platform</a>. In addition, if you can automate provisioning,
the configuration files represent your infrastructure as code, providing
documentation about your system.</p>
<p>VMware vCloud Director supports the <a
href="http://pubs.vmware.com/vcd-51/index.jsp?topic=%2Fcom.vmware.vcloud.api.doc_51%2FGUID-F4BF9D5D-EF66-4D36-A6EB-2086703F6E37.html">vCloud
API</a>, but there was no widely-available tooling to orchestrate this kind of
operation using the API. So we built <a
href="https://github.com/alphagov/vcloud-tools">vCloud Tools</a>.</p>
<h2>Our suppliers currently all use vCloud Director</h2>
<p>The UK Government’s policy is <a
href="https://www.gov.uk/government/news/government-adopts-cloud-first-policy-for-public-sector-it">cloud
first</a> and we procure services through <a title="CloudStore"
href="http://govstore.service.gov.uk/cloudstore/">CloudStore</a> (which uses the
<a href="https://www.gov.uk/how-to-use-cloudstore#g-cloud-framework">G-Cloud</a>
framework). As government, it’s vital that we provide secure, resilient services
that citizens can trust, and we need various assurances from our suppliers to
support that.</p>
<p>There are a number of Infrastructure as a Service (IaaS) platforms that are
able to provide these assurances, but when we procured our current suppliers
last year, the vendors that met our requirements (both technical and
non-functional, eg self-service provisioning, a pay-as-you go cost model) used
vCloud Director.</p>
<p>We very much want to encourage potential new suppliers, but given the
availability in the market, we can be reasonably certain we’ll be using at least
one VMware-based solution for at least the next 12-18 months, and it’s likely
that many other <a href="https://www.gov.uk/transformation">transformation
projects</a> will also be drawing on the same pool of hosting providers. So it’s
definitely worth investing time to make provisioning easy.</p>
<h2>Preventing vendor lock-in</h2>
<p>Having an easy way to provision environments will allow us to move more
easily between vendors that use vCloud Director, which will help prevent
supplier lock-in. But we also don’t want to lock ourselves in to VMware, so we
are taking several steps to guard against that:</p>
<ul>
<li>We access the vCloud API via an Open Source library called <a
href="http://fog.io/">fog</a> which can communicate with multiple types of cloud
providers. If we add different vendor solutions in the future, we can extend our
existing tools.</li>
<li>Within our suite of vCloud Tools, the interaction with fog and the vCloud
API takes place in <a href="http://rubygems.org/gems/vcloud-core">vCloud
Core</a>, which the other tools depend on. This means any changes to use a
different provider would be localised and should not affect the work we’ve done
in the other tools.</li>
<li>We’ve aimed to restrict our usage of VMware to functionality that has
equivalents on other platforms, for example, Load Balancers, Firewalls and NAT,
so we don’t have an infrastructure design that can’t be moved.</li>
</ul>
<h2>Previous iterations of provisioning tooling</h2>
<p>Automation of provisioning is something we’ve been iterating on since we
launched GOV.UK 18 months ago. Prior to vCloud Tools, we used a tool we had
built called vCloud Provisioner. This was really useful when moving onto our
first platform, but because it was built quickly it has a lot of hard-coded
detail about the environment, so it doesn’t help us be flexible when moving
between suppliers. In addition these hard-coded details include sensitive
information, so we can’t share it, meaning it cannot be useful to anyone outside
of the GOV.UK team.</p>
<p>Several members of the infrastructure team worked on subsequent iterations of
the provisioning tooling. These iterations included <a
href="https://github.com/gds-attic/vcloud-box-spinner">vCloud Box Spinner</a>
and <a href="https://github.com/gds-attic/vcloudtools">vcloudtools</a>. However,
the people working on these were building these tools alongside their other
business-as-usual work keeping <a href="https://www.gov.uk/">GOV.UK</a> up and
running, so when they ran into issues, it was difficult to find time to address
them. With the <a
href="https://gdstechnology.blog.gov.uk/2014/03/28/migrating-govuk-infrastructure/">migration
to the new platform</a> looming, we needed to prioritise this piece of work in
order to be able to bring up a new environment quickly, reliably and easily.</p>
<h2>We think vCloud Tools will be more robust</h2>
<p>There are several things we have done to improve the chances of producing
robust, more widely useful tools in this iteration.</p>
<p>We have committed the time and resources to this, forming a small team who
focus entirely on vCloud Tools rather than firefighting or other operations
work, and we are <a
href="https://gds.blog.gov.uk/2012/10/12/coding-in-the-open/">coding in the
open</a>, so we won’t fall into the trap of including sensitive information that
it is later too hard to factor out.</p>
<p>Not only are the GOV.UK team “<a
href="http://en.wikipedia.org/wiki/Eat_one%27s_own_dog_food">eating our own
dogfood</a>” by using vCloud Tools to provision our new hosting environments,
there are two other GDS teams also using the tools, the <a
href="https://gds.blog.gov.uk/2014/01/23/what-is-identity-assurance/">Identity
Assurance Programme</a> and the <a
href="https://www.gov.uk/performance">Performance Platform</a>. Other exemplar
projects have started using the tools, and we have already accepted pull
requests from people we do not know, so there is the beginnings of a community
around the tools. This keeps us from making it too GOV.UK specific and means
that we get a lot of extremely useful user feedback.</p>
<p>And in the meantime we are contributing back to Open Source - in one <a
href="https://github.com/fog/fog/commit/e33ad6265">recent release</a> of fog, a
huge number of the contributions were from the GOV.UK vCloud Tools team (who
were, at that time, <a href="https://github.com/annashipman">me (Anna
Shipman)</a>, <a href="https://github.com/snehaso">Sneha Somwanshi</a>, <a
href="https://github.com/mikepea">Mike Pountney</a> and <a
href="https://github.com/danielabel">Dan Abel</a>).</p>
<p>These factors mean that we feel confident that we are going to produce tools
that will continue to be very useful to us and other teams for some time.</p>
<h2>More will follow</h2>
<p>There will be future posts about particular aspects of what we’ve done and
how, including one soon on how we used vCloud Tools in our platform migration.
If there’s anything you’d particularly like more detail on, please let us know
in the comments below.</p>
<p><em>This post originally appeared on the <a
href="https://gdstechnology.blog.gov.uk/2014/05/07/building-tools-to-provision-our-machines/">GDS
Technology Blog</a></em>.</p>
Running OSS projects2014-04-03T00:00:00+00:00https://www.annashipman.co.uk/jfdi/running-oss-projects.html<p>At <a href="http://www.scalesummit.org/">ScaleSummit</a> last week, I proposed a session on running Open Source projects. I am hoping to move the project I’m working on, <a href="https://github.com/alphagov/vcloud-tools">vCloud Tools</a>, from being <a href="https://gds.blog.gov.uk/2012/10/12/coding-in-the-open/">coded in the open</a> to being Open Source and wanted to get the benefit of the accumulated wisdom. It was an excellent session and I really got a lot out of it.</p>
<p>The executive summary is that if you are trying to build a community, you need to optimise for bringing people into the community, and a lot of the discussion focused around that. Here are some of the main things that I took away from it.</p>
<h3 id="good-documentation-is-crucial">Good documentation is crucial</h3>
<p>Optimise documentation for bringing people into your community. It’s unlikely to already be good for that, as you are so familiar with the code and the project that you don’t realise what someone new to it doesn’t know.</p>
<p>One way to do this is to make it clear that you are happy to answer people’s questions – even hundreds of questions – as long as they follow up by submitting a patch to clarify the documentation. Encourage people to raise bugs against the documentation.</p>
<h3 id="make-it-easy-to-get-involved">Make it easy to get involved</h3>
<p>Make it clear how people can contact you for clarification or with ideas. An IRC channel is a great, if you can make it work, but you need to be aware of certain drawbacks. For example, people tend to expect a quicker response than by email, so you need to make sure people are in it. It will be very quiet at night in the UK.</p>
<p>In a later discussion back at GDS we decided not to do that for this project because there would be a lot of effort to make it work and it would be an added distraction. We definitely don’t want to say “Come to our vCloud Tools IRC channel” if there’s no-one there.</p>
<p>As well as (or instead of) an IRC channel, you should have a mailing list. You can either make it open or make it so that only a subset of people can reply. Someone advised that it wasn’t a good idea to have lots of mailing lists, (*announce*, *discussion* etc). It was also pointed out that even if you’re not writing code, managing the community can easily become a full-time job.</p>
<p>The Perl community does communication well, and the example of <a href="http://www.catalystframework.org/">Catalyst</a> was given – this project has 450 committers and the maintainer has changed 5+ times. Also mentioned was this blog post about communication with newcomers to the project: <a href="http://shadow.cat/blog/matt-s-trout/love-your-idiots/">Love your idiots</a>.</p>
<h3 id="respond-quickly-to-external-contributions">Respond quickly to external contributions</h3>
<p>One of the things that I especially wanted to know about was how quickly you should aim to respond to pull requests. This is particularly an issue in that we are going to be managing this in work time and there will be competing priorities. The general consensus was you need to respond within 24 hours, and it’s acceptable for this not to include weekends if you are clear in your documentation that this is the case. It’s important to note that the response doesn’t have to be a review or a merge, it can be as simple as a note saying “Thanks for this, I am going to review it in a couple of days”.</p>
<p>The most important PRs to pay attention to are the ones from new contributors. Again, optimise for bringing people into your community.</p>
<p>It’s a good idea to have CI run tests on PRs so you can see failures before merging – <a href="https://travis-ci.org/">Travis</a> is good for this as it integrates well with GitHub. However, it was stressed that it is important to review the contribution as well, even if it passes all the tests!</p>
<h3 id="communicate-your-vision">Communicate your vision</h3>
<p>Something else I was particularly interested in is what do you do if you feel a well-meaning PR is pulling the project in the wrong direction? It was suggested that a lot of this comes back to documenting your original vision and being very explicit about what this software is aiming to do.</p>
<p>You need one person or a group of people to be responsible for maintenance, and they are the ones responsible for the vision. Someone gave the example of <a href="http://www.mozilla.org/">Mozilla</a> in the early days who just accepted everything, and eventually had to rewrite the core offering.</p>
<p>But also, take it as it comes. If you as the maintainer don’t think a direction is correct, it’s completely fine for someone to fork the software, and you may later find that the fork is doing the job better. The example of <a href="http://gcc.gnu.org/">GCC</a> was given, where a separate fork existed for about 5 years, and the GCC maintainers eventually realised that the fork was what everyone was using and merged it back in.</p>
<p>Modularity also makes the process easier – having a common core that everyone can agree on is relatively easy, and divergence can be supported with plugins.</p>
<p>It is very important that if you close a PR, you do so with a clear explanation, whether this is due to poor quality or incompatible direction.</p>
<h3 id="dont-be-too-hardline">Don’t be too hardline</h3>
<p>One thing we talked about was not being too strict with external contributors. Sometimes people might not have the experience to know how to write tests, either in general or in your particular test framework, so insisting that a PR must have tests before it can be merged is going to put people off who could have really valuable contributions. Some people said they are very happy to write tests for newbies. Talk to the contributor to find out why they wanted to make those changes and then show them the tests you’ve written, maybe asking them whether it looks like the test cases cover the functionality they wanted to add.</p>
<p>However, changes in functionality should definitely include updated documentation and it is more reasonable to reject a PR for lack of that.</p>
<h3 id="assume-people-have-the-best-intentions">Assume people have the best intentions</h3>
<p>This is great advice for life in general. Even if what they are suggesting looks completely wrong and it’s hard to understand how they could have thought it was the right approach, assume they have the best intentions and proceed accordingly.</p>
<h3 id="issue-tracking">Issue tracking</h3>
<p>We currently manage work on the project internally using <a href="http://www.pivotaltracker.com/">PivotalTracker</a>. My plan was, once we’re ready to make it OSS, we move remaining features and bugs to GitHub issues, and work from that. This was seen as a good idea – it makes it clear to people what we are planning to work on and (ideally!) prevents them from raising issues we already know about. It also has a major benefit – you can Google for the issue.</p>
<h3 id="it-must-be-an-active-project">It must be an active project</h3>
<p>You need to be using the software yourself, otherwise it’s a recipe for abandonware. And it’s good to make this activity clear – related to the above point about issue tracking, if all your activity is on your internal tracker and mailing list and private IRC channel, then it won’t be clear to potential users and contributors that the project is still active.</p>
<h3 id="contributing-guidelines">Contributing Guidelines</h3>
<p>It is important to have contributing guidelines. This wasn’t discussed extensively in the session, but in an extremely helpful follow-up email from <a href="https://twitter.com/bobtfish">Tom Doran</a> he pointed me at a number of great resources which included the <a href="https://metacpan.org/pod/distribution/Catalyst-Manual/lib/Catalyst/Manual/DevelopmentProcess.pod">Catalyst development guidelines</a>. I also heard from <a href="https://twitter.com/georgebrock">George Brocklehurst</a> who pointed me at the <a href="https://github.com/thoughtbot/gitsh/blob/master/CONTRIBUTING.md">contributing guidelines for gitsh</a> and also a really useful page that <a href="http://thoughtbot.com/">thoughtbot</a> have on <a href="https://github.com/thoughtbot/guides/tree/master/code-review">code review</a>.</p>
<h3 id="legal-issues">Legal issues</h3>
<p>Something that was raised as we finished up the session and were leaving was that, especially as Government, we need to be very careful about who owns the code. For example, some people may be working for companies where their contract states that the company owns all OSS contributions.</p>
<h3 id="what-next-for-vcloud-tools">What next for vCloud Tools?</h3>
<p>These considerations and some others mean it will be a little while before we move vCloud Tools from coding in the open to OSS, but it was really useful to have this input which covered a lot of things I hadn’t thought about.</p>
<p>Many other things were discussed, including when you need a standalone website, and what tools are good for versioning documentation, but these were the things I found most useful and relevant. Thanks a lot to everyone who took part in the session, it was very valuable and constructive.</p>
<h4 id="notes">Notes</h4>
<p>Note that ScaleSummit operates under the <a href="http://en.wikipedia.org/wiki/Chatham_House_Rule">Chatham House Rule</a> so none of the comments made in the session have been attributed, and thanks to <a href="https://twitter.com/mattbostock">Matt Bostock</a> and <a href="https://twitter.com/philandstuff">Phil Potter</a> whose write-ups (<a href="http://tech.mattbostock.com/2014/03/23/scale-summit/">Matt’s</a>, <a href="https://gist.github.com/philandstuff/9684513">Phil’s</a>), were a very useful addition to my own notes.</p>
Are you thinking of speaking at a conference?2014-03-09T00:00:00+00:00https://www.annashipman.co.uk/jfdi/conference-speaking.html<p>As co-Programme Chair for <a href="http://www.spaconference.org" target="_blank">SPA conference</a> I often find myself trying to persuade inexperienced or completely new speakers that it is worth doing and that they have something great to say. Here are some of the resources that I send on to them.</p>
<p>Nicely expressed post – you don't have to wait to be an expert in something before you submit a talk: <a href="http://blog.sqisland.com/2012/01/you-can-speak-at-conference-too.html" target="_blank">You can speak at a conference too</a>.</p>
<p>More detail as to why what you have to say is worth hearing, and how to get started. <a href="http://cognition.happycog.com/article/so-why-should-I-speak-publicly" target="_blank">So Why Should I Speak Publicly?</a></p>
<p>The <a href="https://us.pycon.org/2013/schedule/talks/list/" target="_blank">list of accepted talks from last year's PyCon US</a> is a long list of great-looking talks on a variety of subjects to spark ideas of what you might speak about.</p>
<p>If you are nervous about public speaking, this post could really change the way you think about it: <a href="http://seriouspony.com/blog/2013/10/4/presentation-skills-considered-harmful" target="_blank">Presentation Skills Considered Harmful</a>. You are just the UI.</p>
<p>Finally, this is a really useful resource <a href="https://twitter.com/maireadoconnor" target="_blank">Mairead O'Connor</a> sent me last week, containing short posts on all aspects of talk preparation: <a href="http://speaking.io/" target="_blank">speaking.io</a>.</p>
Applying for a job at GDS2013-12-24T00:00:00+00:00https://www.annashipman.co.uk/jfdi/applying-for-a-job-at-gds.html<p>Over the next few months we will be recruiting for a number of roles,
including developers and web operations engineers. Since a civil service
application is a bit different to other jobs you might be applying for, we
thought it was worth explaining what the process involves, with some detailed
guidance on how to write the most unusual part of the application - a statement
of suitability.</p>
<h2>Open and fair process</h2>
<p>There are a number of measures in place to make it more likely that
applications to join the civil service are treated fairly and on their own
merit. You can <a
href="http://www.civilservice.gov.uk/wp-content/uploads/2013/03/Best-Practice-Recruitment-Selection-Guidance-v0.1.doc">read
more about the civil service recruitment and selection guidance on the Civil
Service website</a>, but the main thing you should be aware of is that we try
and ensure objectivity by having consistent criteria that we assess applications
on.</p>
<h2>Consistent criteria</h2>
<p>The job advert will list some essential specialist skills and competencies.
The specialist skills are likely to be what you are familiar with in job
applications, for example for a WebOp, “Experience configuring and managing
Linux servers”.</p>
<p>The competencies are a bit more unusual. They reflect important civil service
behaviours, like leadership and delivering value for money, because we are
recruiting not just technical people, but people to become civil servants. This
helps us build a team of people who are not just great technically, but also
have other crucial skills, like communication and team-working.<br />
<a
href="http://resources.civilservice.gov.uk/wp-content/uploads/2012/07/Civil-Service-Competency-Framework-Feb2013.pdf">For
more information on civil service competencies you can look at this
document</a>. </p>
<h2>Why this matters to applicants</h2>
<p>One of the three documents we ask for in your application is a statement of
suitability, which is extremely important, as it is where you get a chance to
show the panel the evidence for how you meet the essential skills and
competencies described in the job advert. </p>
<p>If a candidate's application does not show evidence of even just one of the
competencies, we are not allowed to invite that candidate for interview. This
may seem a little harsh, but it's actually this kind of rule that tries to
ensure the process is open and fair.</p>
<p>Based on this, we can see how the statement of suitability is so important in
the application. Your CV is unlikely to provide quite the right sort of
information on its own, as CVs tend to be a collection of achievements and
responsibilities. The Statement of Suitability is your opportunity to fill out
the gaps in your CV and explain you have the experience to do a great job at
GDS. You must remember to give examples, as that is what we are looking for in
the application.</p>
<h2>How to write a good statement of suitability</h2>
<p>The main thing required in the statement of suitability is that you
demonstrate the essential skills and competencies asked for. Here is an
example.</p>
<p>We recently advertised for Web Operations Engineers (and will be doing so
again - watch this space!). One of the essential competencies we are looking for
is evidence of how you "coach and support colleagues to take responsibility for
their own development (through giving accountability, varied assignments and
on-going feedback)".</p>
<p>If you have managed a team, it should be fairly straightforward to think of
an example. But if you haven't, there are other ways you can provide this
evidence.</p>
<ul>
<li>if you have done any person management, you might be able to give an example
of how you have assisted colleagues in writing their appraisals</li>
<li>you might be able to give an example of doing code reviews, or feedback on
pull requests, or helping someone learn how to script a manual process</li>
<li>you might have an example from non-work development activities, for example
if you volunteer at a Code Club or you have mentored someone in an Open Source
project</li>
<li>your examples don't have to be about something you did while working in a
technical role, for example if you coach a netball team you might be able to
give an example from that</li>
</ul>
<p>The point here is that we feel an ability to coach and support colleagues is
essential to being successful in the role, and we are looking for evidence that
you have done this in some context before.</p>
<h2>A word on how to phrase your answers</h2>
<p>It's good to give detail, but your answers don't have to be really long. The
best thing to do is think of the example you want to use and give us the
essential information.<br />
It may be useful to think of the CAR approach:</p>
<ul>
<li>Context: what the situation was and what you were trying to achieve</li>
<li>Action: what you did</li>
<li>Result: what the outcome was.</li>
</ul>
<p>If you've covered all these points, you will have structured your example to
give us all the information we need.</p>
<h2>How to structure the statement</h2>
<p>You might find it helpful to list each of the essential skills and
competencies as headings and give an example under each, or you may prefer to
write it in a letter or essay format. The structure is not important - just make
sure that you cover all of the essential competencies asked for.</p>
<h2>The other documents</h2>
<p>The other documents we require are your CV and CV cover sheet. There's
nothing different about how you should structure your CV compared to applying
for a job in the private sector.<br />
The CV cover sheet is a form which requires you to fill in some straightforward
details so that we know how to contact you. Download this from the very bottom
of the job application page, and include it with your application. If you are
not already a civil servant you only need to complete three parts of this form.
For existing civil servants, you need to fill in more sections of the CV cover
sheet - your line manager should be able to help you with this.</p>
<h2>Interview stage</h2>
<p>Once we have sifted all of the applications we invite some of the applicants
to interview. All candidates are asked a consistent set of questions for that
role, and you are likely to also have to do some kind of coding or whiteboard
exercise. If you are called for interview, you will get more information about
what to expect in advance.</p>
<p><em>This post originally appeared on the <a
href="https://gdstechnology.blog.gov.uk/2013/12/24/applying-for-a-job-at-gds/">GDS
Technology Blog</a></em>.</p>
How Anonymous Submissions Affected Gender Balance at our Conference2013-11-17T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-anonymity-affected-gender-balance.html<p>Mainly inspired by <a href="http://2012.jsconf.eu/2012/09/17/beating-the-odds-how-we-got-25-percent-women-speakers.html" target="_blank">this article</a>, I proposed that we make submissions anonymous for <a href="http://www.spaconference.org/spa2013/" target="_blank">SPA2013</a>. The committee (four of us) discussed what we thought the outcome would be for some time, before deciding to just try it and see how it worked out. And here's the answer.</p>
<h3>The conference</h3>
<p>Firstly, a bit about SPA. It stands for Software Practice Advancement, and it's a very interactive conference – you are unlikely to be sitting in a talk. You'll be writing code in a new language, or doing a group exercise in a process track, or taking part in a goldfish bowl discussion, or actively learning in another way. It's a great conference, and if you're interested in checking it out we are holding a free one day taster with some of the best sessions from this year in Cambridge on 23rd November. <a href="http://www.spaconference.org/spa2013/minispa-2013.html" target="_blank">MiniSPA</a>. It's free!</p>
<p>Enough pitching. I've attended SPA three years running and became co-Programme Chair for the 2013 conference, and while I think it's very good, one thing I think would improve it is more women leading sessions. I would also like it if we could get a greater diversity of speakers in other ways, but I thought I would focus on women for starters.</p>
<h3>Stats</h3>
<p>The first year I attended, <a href="http://www.spaconference.org/spa2011/programme.html" target="_blank">SPA2011</a>, there were 28 sessions, with a total of 43 session leaders between them. Of those 43 session leaders, 38 were men and the remaining 5 were women.</p>
<p>Other interesting things I noticed were that four people presented more than one session at the conference – all four were men. Only nine sessions were led by just one person, and of those nine, eight were men.</p>
<div class="pretty-code-sample">
<h3>2011</h3>
<p>5/43 = 12% of the session leaders were women.</p>
<p>1/9 = 11% of the solo session leaders were women.</p>
<p>0/4 = No women ran more than one session.</p>
</div>
<p>The <a href="http://www.spaconference.org/spa2012/programme.html" target="_blank">next year</a>, there were 30 sessions and 42 presenters. Four were women. Again, the five presenters running more than one session were all men. Ten sessions were led by a single presenter – 8 men and 2 women (one was <a href="http://www.spaconference.org/spa2012/sessions/session412.html" target="_blank">me</a>!).</p>
<div class="pretty-code-sample">
<h3>2012</h3>
<p>4/42 = 10% of the session leaders were women.</p>
<p>2/10 = 20% of the solo session leaders were women.</p>
<p>0/5 = No women ran more than one session.</p>
</div>
<h3>How we did anonymity</h3>
<p>At SPA we have three stages after submission of proposals.</p>
<ol>
<li>A feedback stage, where people offer advice to help the session submitter improve the submission (which can be
edited up until the deadline).</li>
<li>A review stage where at least three reviewers
grade a submission according to whether they think it should be included in the
programme.</li>
<li>A programme meeting, when we pull the reviews together and decide
on the programme.</li>
</ol>
<p>In previous years, the presenters' names appeared alongside their submissions all the way through. This year, we made them anonymous right up until we had put together a draft programme, and then we did the big reveal, just to check that we hadn't scheduled one person to do several sessions at the same time, for example.</p>
<h3>Results</h3>
<p>So did it make a difference?</p>
<p>At <a href="http://www.spaconference.org/spa2013/programme.html" target="_blank">SPA2013</a>, we had 27 sessions with 46 presenters, 40 men and 6 women. Of those, two were repeat session leaders (both men) and 10 sessions had single leaders, of whom one was a woman.</p>
<div class="pretty-code-sample">
<h3>2013</h3>
<p>6/46 = 13% of the session leaders were women.</p>
<p>1/10 = 10% of the solo session leaders were women.</p>
<p>0/2 = No women ran more than one session.</p>
</div>
<p>So no, not really.</p>
<p>That is – to the number of women speaking. A lot of the conference feedback praised the variety of sessions and topics, more so than in previous years. So we did something right. I also don't know whether it made any difference to the number of women who submitted, as I don't have that data.</p>
<p>But there were two other very interesting outcomes.</p>
<h3>Lack of positive bias is good for everyone else</h3>
<p>The first unintended – but good! – outcome was that people new to speaking at
the conference felt much more able to submit.</p>
<p>A very representative comment from the survey I conducted was:</p>
<div class="quote">"Since I haven't spoken at SPA before, I expect being anonymous helped me compete with regulars."</div>
<p>And correspondingly, the reviewers had a lot of very similar comments. Here's
one that sums up most of them:</p>
<div class="quote">"We used to have too many reviews along the lines of 'a bit weak but I know Alice and Bob and they'll do a good job'. They didn't always do a good job and it resulted in too many repeat speakers, which I think new potential speakers find off-putting."</div>
<p>Many of the reviewers admitted they didn't actually like anonymity but realised it helped prevent bias. Of course, we are generally not aware of our subconscious biases *against* people, but one thing that anonymity made reviewers aware of was our tendency to be biased *towards* people we know. But of course, that kind of positive bias means that someone merely unknown will have to work harder to get their session considered.</p>
<h3>Positive discrimination can be off-putting</h3>
<p>The second benefit of the anonymity became apparent in a scenario that I hadn't even considered. One submitter proposed a session about probabilistic data structures, which looked excellent, but her language of choice was Haskell, and most of the feedback focused on attempting to get her to change to a more widely used language. Based on that, she concluded that the talk probably wasn't right for SPA, and assumed that she would later hear it had been rejected.</p>
<p>In fact the talk was very popular with the reviewers, receiving top grades from all of them, so it was a shoe-in for the conference. But when we contacted her to let her know she was in the programme, she initially said she was no longer able to give the talk. A day or so later, she got back in touch to say – to our relief – that she actually was available, and explained her initial reluctance:</p>
<div class="quote">"Truth is the session feedback concluded that the idea wasn't right for the conference, so I inferred that you must've accepted the session because I'm female, and had a bit of an emotional reaction; like any human I want to have opportunities based on my own merits."</div>
<p>In actual fact, this was not what had happened, as neither I nor any of the reviewers knew who she was. Her session had been accepted purely on the merits of the submission. And a massive advantage of the anonymity we had put in place meant that we could claim with certainty and evidence that we were not making decisions in a biased way, either for or against her submission based on her gender.</p>
<h3>A possible objection</h3>
<p>Some people surveyed felt that it might be appropriate to remove the anonymity at the review stage. Ability to present content can be at least as important as the content itself, and people felt the best way to get this information was from knowing who the speaker is. Here is a representative comment:<p>
<div class="quote">"I think that the knowledge of a presenter's past delivery can be a big differentiator when you get submissions of similar quality."</div>
<p>But really, this is just another way of saying we want to know when people are well-known – we want to put Alice and Bob in because they always do a good session.</p>
<p>Yes, a speaker's delivery and structure makes a huge difference to how well a session works. However, at SPA we assign shepherds to first-time presenters (or anyone who wants one) and they can be very helpful – my excellent shepherd <a href="http://blog.caplin.com/author/adamicaplincom/" target="_blank">Adam Iley</a>, among other things, arranged a run-through at his workplace to allow me to practice my session with an unfamiliar audience.</p>
<p>So if we do accept a promising session where we don't know that the speaker is good, we might be able to help with that; and conversely, I tend to think that a good presenter should be able to write a good submission so that will shine through.</p>
<h3>Next!</h3>
<p>In any case, the committee felt that on balance it was positive, so we are continuing with anonymity throughout the whole process this year.</p>
<p>Why not get involved? Submit a session, or get involved in giving feedback on sessions, or both!</p>
<p>Check out the <a href="http://www.spaconference.org/spa2017/lead-a-session.html" target="_blank">call for proposals</a>, and have a
look at the previous programmes I've linked to above for an idea of the kinds of
sessions we like.</p>
<p>And if you have any questions, or want to get involved in giving
feedback, please do <a href="http://www.spaconference.org/spa2014/organisers.html" target="_blank">get in touch</a>.</p>
Creating a Puppet module2013-09-09T00:00:00+00:00https://www.annashipman.co.uk/jfdi/creating-a-puppet-module.html<p>I learnt three very interesting things recently, courtesy of the brilliant <a
href="https://dan.carley.co/" target="_blank">Dan Carley</a>.<p>
<ol>
<li><a href="#git-filter-branch">git filter-branch</a></li>
<li><a href="#generate-module">how to create a puppet module using our puppet module skeleton</a></li>
<li><a href="#shopt">shopt -s dotglob</a></li>
</ol>
<p>While creating the Puppet manifests for the GOV.UK mirror, we realised we
needed a module that was part of the GOV.UK Puppet, so our task was to pull it out into a standalone module and
have both GOV.UK and our new <a
href="https://github.com/alphagov/govuk_mirror-puppet" target="_blank">GOV.UK
mirror</a> use it.</p>
<p>The first step in creating a new module is to pull it out into its own git
repository.<p>
<p>First, clone the repository the module is currently in. (Extra learning here:
you don't need to clone it from GitHub, you can just git clone the existing
repository.) You want <span class="code-sample-inline">--no-hardlinks</span> so it's entirely separate:</p>
<div class="code-sample">
git clone --no-hardlinks puppet puppet-auditd
</div>
<p>At this stage our puppet-auditd repo still points at our puppet repo, so we
want to remove the remote:</p>
<div class="code-sample">
git remote rm origin
</div>
<p><span id="git-filter-branch">Now</span> we want to get rid of everything that's not our module and make our module
the root, and this is where we use <a
href="http://teach.github.com/articles/lesson-filter-branch/" target="_blank">git
filter-branch</a> (update 2014-04-12: GitHub have removed this without a redirect – tsk! – but you can find a historical copy <a href="https://github.com/github/training.github.com/blob/4cdc8c9c72ba3b0a82612bd0f709d679f3020b01/technology/_posts/2001-01-01-lesson-filter-branch.md" target="_blank">here</a>):</p>
<div class="code-sample">
git filter-branch --subdirectory-filter modules/audit-d HEAD
</div>
<p>The next thing we want to do is create the files in the framework around the
subdirectory
to make it a Puppet module, for example the Modulefile. For this we can use our <a
href="https://github.com/gds-operations/puppet-module-skeleton"
target="_blank">Puppet module skeleton</a>. However, we don't want
to directly follow the instructions there as we already have a lot of the
classes we need. Instead of creating it from scratch as per those instructions,
we <span id="generate-module">generate our new module stucture</span>:<p>
<div class="code-sample">
puppet module generate gdsoperations-auditd
</div>
<p>and then then we copy over the generated files.</p>
<p>However, a normal recursive copy won't work:<p>
<div class="code-sample">
cp -r A/* B
</div>
<p>This will only copy non-dotfiles, and we want the dotfiles as well, like
.gitignore.</p>
<p>However, attempting to copy the dotfiles as well:</p>
<div class="code-sample">
cp -r A/.* B
</div>
<p>leads to hilarious results. <span class="code-sample-inline">..</span> is a
directory too, so the above command copies all those contents recursively as
well. Try it. It's most entertaining. But it's not really what we want.
<p><span id="shopt">What</span> we can do is toggle what is included in <span class="code-sample-inline" >A/*</span> for this shell. The
documentation is <a
href="http://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html"
target="_blank">here</a>, but this is what we need:
<div class="code-sample">
shopt -s dotglob
</div>
<p>In order to check what is going to be copied now you can:<p>
<div class="code-sample">
echo A/*
</div>
<p>and if you like it:<p>
<div class="code-sample">
cp -r A/* B
</div>
<p>The next step is to go through the files to work out which modifications to
keep and which ones are not relevant.</p>
<img src="/img/files_changed.png">
<p>Another tip from Dan: vimdiff is really useful for this task. I won't go into
how and what we decided; you can see that <a
href="https://github.com/gds-operations/puppet-auditd/compare/c140536...cad9a94"
target="_blank">here</a>.</p>
Some Regex in the form of a Picture!2013-08-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/some-regex-in-the-form-of-a-picture.html<img src="/img/regex.jpg" />
Where Do I Start?2013-07-06T00:00:00+00:00https://www.annashipman.co.uk/jfdi/where-do-i-start.html<p>The first day at a new job is always a bit nerve-wracking. In a recent
session at <a href="http://www.spaconference.org/spa2013/" target="_blank">SPA
2013</a>, we came up with some things a developer joining a new team can do to get up to speed quickly.</p>
<img src="/img/where_start.jpg" />
<h3>Whiteboard an architecture diagram</h3>
<p>This is a brilliant way to get a quick overview of how the pieces of
your new project fit together, and can be tailored to whatever level of
understanding you (or the person you are talking to) has. It's a good way to
meet people on other teams, when you're looking to get more detail on some of
the parts of the diagram. And your fresh perspective can often be very useful to the person taking you through the architecture.</p>
<p>However, it can sometimes be hard to know who the right person is to ask, and
it can also be hard to get the right level of detail.</p>
<h3>Facilitate a retrospective</h3>
<p>This is the fastest way to find out what's going on in a team, and beacuse
you're new, you will essentially be an external facilitator which is really
useful to the team. It's also a great way to find out what's going on in other teams once you've settled in.<p>
<p>However, it could be challenging if your new place of work is not used to
agile working, and of course you do have to be the kind of person who has the
confidence to faciliate a retrospective for a bunch of people you don't know
yet.</p>
<h3>Start putting a glossary together</h3>
<p>...with the aim of putting it on the wiki.</p>
<p>This will give you a head start on understanding the concepts that the rest of team take for granted – and has the added advantage that you may hit on areas of vagueness or disagreement that can be straightened out.</p>
<p>It can however be hard to get started with this, and to work out what to focus on, and in some teams it might be a challenge to find anyone who
understands them all.</p>
<h3>In Summary: Do something, document something</h3>
<p>People have different ways of learning and sometimes the explanations you get
in your first week can lead to information overload and be hard to take in. So it can be
useful to ask more directed questions, for example in pursuit of added detail
for an architecture diagram, or to draw out actions for a retrospective.</p>
<p>We had some other ideas as well:</p>
<li>Pair program as much as you can, with different people</li>
<li>You could think about shadowing someone for a morning or a day</li>
<li>Talk to other teams - the teams that particularly can give you insight into
what you need to know are the testing team, the support team and the ops
team</li>
<li>Find out where people go for lunch and wangle yourself an invitation!</li>
</ul>
<p>We also stole some ideas from other teams, like if the team's processes are
not documented you could offer to do that as a way to gain understanding. For
example, how well documented are the new joiner instructions? You are in a great
place to improve those, and a big win would be automating some part of that.
Also, meeting the users is a fantastic way to understand the business, so if
that's not part of your induction you could arrange that.</p>
<p>Thanks to <a href="https://twitter.com/andylongshaw" target="_blank">Andy
Longshaw</a>, <a href="https://twitter.com/eoinwoodz" target="_blank">Eoin
Woods</a> and <a href="http://rozanski.org.uk/" target="_blank">Nick
Rozanski</a> for organising the session.</p>
Roof Bug-fixing2013-05-21T00:00:00+00:00https://www.annashipman.co.uk/jfdi/roof-bug-fixing.html<p>I often find myself viewing life through the lens of software development, a bit like when you play too much Tetris and all your friends turn into shapes. But recent events have made me think: maybe other things actually would be better if they were more like software development?</p>
<p>Recently, I had a leak into my flat. By "recently" I mean it's been ongoing for four and a half months, but that's not what this post is about. This is about applying software development ideas to other fields, instead of the other way around. We talk about <a href="http://manifesto.softwarecraftsmanship.org/" target="_blank">"software craftsmanship"</a> – I'm thinking about the opposite, as applied to, for example, roofing.</p>
<p>I first noticed the leak as I was about to go on holiday before Christmas. It rained very heavily, and literally hours before I was due to leave, I saw water bubbling up between the laminate floorboards in the hall. I pulled up the vinyl on the bathroom floor and then took the side of the bath off, and found that my bathroom was about an inch deep in water.</p>
<p>Cut forward a fortnight to when the housing association manage to send out a surveyor to investigate. He tells me what is causing the problem: the flat above and to the right has – without permission – erected some kind of structure on their balcony, and this has caused damage to the brickwork, so when it rains water is seeping into my flat through the holes in their balcony.</p>
<img src="/img/balcony.jpg" />
<p>At this stage, it's a bit of a mystery to me how that works. And why shouldn't it be? I'm not a roofer. Or a surveyor, or a builder, or an architect. I have only the vaguest notion of how buildings are put together, and I don't think to ask for an explanation of how the water is filling up my flat. Apparently, the Offending Neighbour has drilled a hole to put up the mystery structure, and the hole is the problem. They are going to asphalt over it. And they do so.</p>
<p>All well and good, and the insurance company come round to measure up the flat to fix the damage. Except the day they come to do this, it rains heavily, and once again, I see water bubbling up through the floor in the hall. They have asphalted over the hole in the balcony, but this hasn't fixed the leak!</p>
<p>A builder comes to my flat to discuss it with me. This is where it starts to get interesting. The builder has already looked at the Offending Neighbour's balcony and now he wants to look at my flat. But he can't see how the pieces fit together. Eventually, I have to take him outside to point at the balcony to demonstrate that water coming from the balcony at the point at which it intersects with my flat would lead to damp at exactly the spot we see it, as per the diagram above.</p>
<p>This is when I first start to think of this in terms of a bug. Here is the point at which you need to look at the bigger picture. We're not talking about a bit of mistaken logic in a Ruby program affecting the output of that program. We're talking about a timeout two layers away. The manifestation of the problem is a long way from the cause, and you really need a view of the bigger picture to be able to reason about it.</p>
<p>So the builder goes away, and finally (after calls and emails and threats and complaints) the housing association get back to me and tell me that they are going to put something called <a href="http://en.wikipedia.org/wiki/Flashing_(weatherproofing)"target="_blank">flashing</a> on the point at which the wall of my flat meets the wall of ON's balcony. This, they tell me, will definitely fix the problem.</p>
<img src="/img/flashing.jpg" />
<p>So this makes sense to me. At this point, I've got an idea of the bigger picture, though the details of the water seeping through the brickwork and then somehow bubbling up through the floor are somewhat hazy to me. But I do have one strong opinion at this point: I want to be sure this time that they have really fixed it. At this point, I conceive of the idea of a <a href="http://www.universalroof.com/orlando-roofing/water-testing.html" target="_blank">water test</a>.</p>
<p>You see, when we find a bug in our software, we try and recreate it. Then we know when we've fixed it. And something I was starting to notice here was that this wasn't happening. It was as if I had noticed the timeout, and made a configuration change in the general area, then marked the story as done and walked away. We don't do that, with software. And yet this situation that was making my life pretty inconvenient – three months in, my flat was covered in mould, smelt of damp and all my furniture was in one room while I waited for the leak to be be fixed – was being dealt with by people who seemed to be making general stabs in the direction of the problem, without any kind of theory or analysis.</p>
<p>Of course, I didn't quite realise that – you expect people to be professionals, whatever the job is. But I was sure I wanted them to do a water test.</p>
<p>But getting them to do this was pretty hard, even though it seemed completely obvious to me. What's the problem? Stand on the balcony with a hose, see if the problem is fixed! At one point, I was told it wasn't "feasible". I started to wonder if I was being too idealistic and actually a building was more like a huge legacy codebase where you may not even have access to all the libraries. Maybe I'd just have to accept a best guess instead of a rigorous approach.</p>
<p>Finally, four months in, I managed to persuade them to do it. The head of repairs at the housing association came round to instruct the roofer, but even as he did this he was complaining that the water test was pointless, a waste of time, as it was due to rain today anyway and that would test it. He didn't really grasp the idea that a water test is a controlled test – it's a way to control the input so as to work out whether the output is what you'd expect were your conclusions correct. Rain – apart from being unpredictable – is not controlled. For a start, it rains all over the building at once.</p>
<p>But then the roofer came round, and he was a true software developer.</p><p>We went up to the balcony together and he explained to me what he was going to do. Firstly, he was going to hose down on the balcony, not going over the edge. This was to test the flashing.</p>
<p>There is a gap between ON's fence and the wall of my flat. It seemed to me that the gap was too low and my suspicion was that rainwater was overflowing through the gap and thus soaking into my wall.</p>
<img src="/img/top_view.jpg" />
<p>However, the roofer explained to me that this was the design of the balcony, to prevent flooding. If it rained heavily, the water would flow along the gully and into the drain, and the gap was to prevent the balcony flooding if the rain was too heavy.</p>
<p>However, secondly, he explained, he was going to hose down specifically through the gap and over the side of the wall of my flat so we could see what would happen if the balcony did flood. The reason he was doing these things separately, he told me, was so he could isolate the cause of the problem. If he did both at once, we wouldn't know what the specific problem was.</p>
<p>Yes! Exactly! This is why we don't make the arbitrary configuration change and correct the logic in the Ruby program at the same time, because then how do we know? This man was speaking my language!</p>
<p>The first thing took him about thirty seconds to discover. They turned the hose on the flashing, the water ran down the gully as planned and then – that was it. The drain was completely blocked. It took less than a minute for the balcony to flood and the water to start pouring through the gap and down my wall. Thus demonstrating another benefit of methodical testing – you surface things you might have assumed to be different.</p>
<p>Later, when unblocking it, the roofer told me it must have taken years to get to that state. One might have thought that ON would have reported it at some point in those years. But why would she? She may not even have noticed – presumably she doesn't hang around outside when it's raining. It had not occurred to any of the previous investigators of this problem to check the drain. And while it may seem an obvious thing to have checked, one often overlooks the obvious, and that is why testing is good.</p>
<p>The second thing took this software genius of a roofer a few more minutes to discover. After unblocking the drain, he hosed down the side of the my building and at this point I found that there was water coming in under the bath again. He looked closer at the building and saw, under where the balcony joined my wall, a gap.</p>
<img src="/img/gap.jpg" />
<p>Having hosed the wall, he had seen that water ran into that gap and from there, the easiest escape would be into my flat, rather than back out. By attempting to recreate the problem, he identified the solution.</p>
<p>So he unblocked the drain, and he filled in the hole with cement. And then, as if he had not covered himself in glory enough, he told me he was only 95% certain that this would solve the problem, whereas the previous fixes, I had been assured, were the solution. He knows a building is a legacy system. But he has the software methodology approach.</p>
<p>There's more learning here as well – for example, to not assume that you won't understand a problem that isn't in your field – but the main thing I took from it was this: everything would be better if it was a bit more like software development. Craftsmen should be more like software developers.</p>
<img src="/img/fixed.jpg" />
Upgrading to 12.04...2013-05-06T00:00:00+00:00https://www.annashipman.co.uk/jfdi/upgrading-to-12-04.html<p>I needed to write some code. Thanks to <a href="http://georgebrock.com/images/thumbsup.gif" target="_blank">George Brocklehurst</a>, I was no longer content to do this on my personal laptop using gedit – I required Vim.</p>
<p>But I could not upgrade Vim!</p>
<div class="code-sample">
Failed to fetch http://gb.archive.ubuntu.com/ubuntu/pool/main/v/vim/vim_7.2.330-1ubuntu4_amd64.deb 404 Not Found [IP: 91.189.92.201 80]<br />
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?<br />
</div>
<p>OK, maybe it was unrelated, but it was time to face the inevitable. I was on Ubuntu 10.10. I needed to upgrade. <a href="#hard-earned-learnings">Skip to what I learned from this sorry saga.</a></p>
<p>Great, so all I needed to do was update to next package from the Update Manager, right?</p>
<div class="code-sample">Error during update<br />
A problem occurred during the update. This is usually some sort of network problem, please check your network connection and retry.<br />
, W:Failed to fetch http://extras.ubuntu.com/ubuntu/dists/natty/main/binary-i386/Packages.gz 404 Not Found<br />
, E:Some index files failed to download, they have been ignored, or old ones used instead.<br /></div>
<p>Ah. Wrong. I left it so long that it was <a href="http://askubuntu.com/questions/226906/how-to-upgrade-ubuntu-unsupported-10-10-to-11-04-then-ultimately-to-12-10" target="_blank">no longer supported</a>.
<p>Right, so I couldn't go to 11.04. But actually, I didn't want 11.04, I wanted 12.04. So can I not just go directly there?</p>
<p>Well, no, as it turns out. Or maybe you can. I still don't know. But threads like <a href="http://askubuntu.com/questions/34430/can-i-skip-over-releases-when-upgrading" target="_blank">this</a> suggested that I couldn't go directly to 12.04 without doing a fresh install and losing my files. I didn't fancy that, so I thought I'd go the long way round.</p>
<p>Since I was out of time to upgrade via the Update Manager, I needed to download the <a href="http://old-releases.ubuntu.com/releases/natty/" target="_blank">11.04 ISO</a>, confirming that it was, as I thought, 64-bit:</p>
<div class="code-sample" id="64">
uname -m<br />
x86_64<br /></div>
<p>and then save the ISO to a CD. Having not done this before, I found <a href="https://help.ubuntu.com/community/BurningIsoHowto" target="_blank">this</a> useful. You want the CD to have multiple files and folders, not just the ISO as one file.</p>
<p>Great. So then you stick the CD in and just follow the instructions. Like <a href="http://www.liberiangeek.net/2012/04/upgrade-to-ubuntu-12-04-from-11-10-using-ubuntu-cd-dvd/" target="_blank">these</a>. Fine. No problem.</p>
<p>Except... problem. <a href="http://www.liberiangeek.net/wp-content/uploads/2012/04/upgrade_cd_percise_4.png" target="_blank">This bit?</a> That hung there for HOURS. If you click the arrow you can see the command it is hanging on, but Googling that on another computer (which I had by now switched on in order to continue with my life) gave me nothing.</p>
<p>I'm telling you this, even though I'm embarrassed: I killed it. It had been hanging there for hours! I couldn't take it any more!</p>
<p>And funnily enough, that turned out not to have an ideal outcome... it would not boot. On the plus side, it's dual boot and Windows seemed fine, though at this stage, I was pretty sure I could "fix" that too:</p>
<blockquote class="twitter-tweet"><p>Current status <a href="http://t.co/mgSQUIBrUY" title="http://xkcd.com/349/">xkcd.com/349/</a></p>— Anna Shipman (@annashipman) <a href="https://twitter.com/annashipman/status/318362562385309696">March 31, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Fine, never mind, all my files etc were backed up, it's no biggie. In fact, I could just install Ubuntu 12.04 at this point? Since it appeared that I may well have lost all my files.</p>
<p>Except somehow, no. The other computer I was using was Windows, and – who knew? – it wasn't so easy to get the ISO onto a CD. After some trying, I gave up.</p>
<p>Next plan was to install 11.04 from the disc I actually had. And, to my joy and incredulity – THIS WORKED!</p>
<blockquote class="twitter-tweet"><p>Omg! So I've managed to upgrade to 11.04. I mean, yeah, I was aiming for 12.04, and OK, so I've lost all my files but, y'know, go me!</p>— Anna Shipman (@annashipman) <a href="https://twitter.com/annashipman/status/318384505117372416">March 31, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>And then, amazingly, the upgrade to 11.10, and then 12.04, could be done via the Update Manager. Finally, six hours after I'd embarked on this foolish mission I was able to report back:</p>
<blockquote class="twitter-tweet"><p>Made it to 12.04! Hurrah! All I need to do is restore my lost files from backup and then I can get on with what I planned to do 6 hrs ago...</p>— Anna Shipman (@annashipman) <a href="https://twitter.com/annashipman/status/318423522768723968">March 31, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>...shortly followed by...<p>
<blockquote class="twitter-tweet"><p>Current status (finally!) <a href="http://t.co/vJVhFTg6Y6" title="http://i.imgur.com/VhlQK.gif">i.imgur.com/VhlQK.gif</a></p>— Anna Shipman (@annashipman) <a href="https://twitter.com/annashipman/status/318660522130149376">April 1, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<h3 id="hard-earned-learnings">So what have I learned?</h3>
<ol><li>How to check your system is <a href="#64">64-bit</a>.</li>
<li>How to embed tweets into blog posts (click the 'More' button to the right of the 'Favourite' button and all will be revealed).</li>
<li>Not to put off the upgrade until such time that it's no longer supported and you have to go through all this!</li>
<li>ALWAYS be backing up your files. As Jeff Atwood said, "until you have a backup strategy of some kind, <a href="http://www.codinghorror.com/blog/2009/12/international-backup-awareness-day.html" target="_blank">you're screwed</a>, you just don't know it yet". This could've been much worse for me.</li></ol>
<p>Happy <a href="http://i.imgur.com/VhlQK.gif" target="_blank">programming</a>!</p>
Partitioning!2013-04-24T00:00:00+00:00https://www.annashipman.co.uk/jfdi/partitioning.html<p>Today I learned from the awesome <a
href="https://twitter.com/rjw1" target="_blank">bob</a> how to add storage to a box,
including partitioning, mounting, and other things I'd heard of but had never
done myself.</p>
<p>We had to add some storage to the elasticsearch boxes. We generate a lot of logs
and they all go to elasticsearch. By default when we provision servers we give
them 32GB but this wasn't enough. So here's how to do it:</p>
<ol>
<li>Add the extra disk through our virtualisation platform's control panel.</li>
<li>Reboot the box so that it sees the new disk.</li>
<li>Partition the new disk. There are many ways to partition a disk - we used <a
href="http://en.wikipedia.org/wiki/Cfdisk" target="_blank">cfdisk</a>, which has a GUI.
<div class="code-sample">cfdisk /dev/sdb</div>
This opens the GUI and then you tab through it to select options. We just
accepted all the defaults, so the only time we had to change it was at the end,
to select [write].
We did consider making the type Linux LVM as this would have meant we could extend it
later, but in this case we just accepted the default type of Linux.<br />
This means that as well as /dev/sdb we have /dev/sdb1. (Extra learning points
here - disks are added alphabetically. So the existing disk was sda. If,
subsequent to this, we add another disk, it will be sdc.)</li>
<li>Before we can mount the disk we need to give it a filesystem. We gave it <a
href="http://en.wikipedia.org/wiki/Ext4" target="_blank">ext4</a> because "everything uses ext4
these days" (bob). If you don't have a good reason for picking one of the others, then
use ext4.
<div class="code-sample">mkfs.ext4 /dev/sdb1</div></li>
<li>Now we mount the new disk to a temporary directory:
<div class="code-sample">mkdir /mnt/tmp<br />
mount /dev/sdb1 /mnt/tmp</div></li>
<li>At this point we need to stop elasticsearch and also disable puppet (otherwise it will just start elasticsearch again on its next run):
<div class="code-sample">service elasticsearch-logging stop<br />
puppet agent --disable</div></li>
<li>Now we need to copy the elasticsearch indices to the new disk. Because the indices are just files you can rsync them, as long as elasticsearch has been stopped, which it has.
<div class="code-sample"> rsync -av --stats --progress /mnt/elasticsearch/*
/mnt/tmp</div>
The -v flag above is 'verbose', and in addition, the stats and progress options
give some more noisy output while the rsync is going on. Often, when using rsync
you would want to add -z, but since this is just from one disk to another and
not going over the wire then there's no need here. Finally, -a is 'archive'. As
the man page helpfully notes, this is "same as -rlptgoD (no -H)". I will leave
that as an exercise for the reader.<br />
This stage takes some time.</li>
<li>When it's finished, we did two things, just as a (slightly paranoid) check
to see if it worked:
<ol><li>Run the same command again. If it has successfully rsynched everything, the two will be the same so it will output immediately.</li>
<li><a href="http://en.wikipedia.org/wiki/Du_(Unix)" target="_blank">du
-sh</a> to check that /mnt/elasticsearch and /mnt/tmp are the same
size.</li></ol></li>
<li>Instead of mounting the new disk in the same way as step 5 above, we can edit <a href="http://en.wikipedia.org/wiki/Fstab" target="_blank">/etc/fstab</a>. We add this line:
<div class="code-sample">/dev/sdb1 /mnt/elasticsearch ext4 defaults 0
0</div>
We should have thought about using <a
href="http://puppetlabs.com/puppet/what-is-puppet" target="_blank">Puppet</a> here, as doing it like this means there are parts of the infrastructure that are not code.</li>
<li>Next, we unmount sdb1 from /mnt/tmp:
<div class="code-sample">umount /mnt/tmp</div></li>
<li>Then, we move the old elastic search directory and create a new one:
<div class="code-sample">mv /mnt/elasticsearch /mnt/elasticsearch-old<br />
mkdir /mnt/elasticsearch</div></li>
<li>Now mount the sdb1 to the new elasticsearch. Because of step 9 above, we can pass either the mount directory or the directory to be mounted; we do not need both. This has the added advantage of testing our fstab setup.
<div class="code-sample">mount /mnt/elasticsearch</div></li>
<li>All done! Now we need to:
<ol>
<li>Restart elastic search</li>
<li>Re-enable puppet</li>
<li>Delete /mnt/tmp and /mnt/elasticsearch-old</li>
</ol></li></ol>
Roof Hacking2013-03-24T00:00:00+00:00https://www.annashipman.co.uk/jfdi/roof-hacking.html<p>Two weeks ago I was at the excellent <a href="http://devfort.com/" target="_blank">/dev/fort</a>. More on that another time; all I'll say now is that we were making a website in a <a href="http://www.landmarktrust.org.uk/search-and-book/properties/fort-clonque-7423" target="_blank">fort</a>.</p>
<p>On one feature I was pairing (actually, tripling...) on some Django with the awesome <a href="https://twitter.com/georgebrock" target="_blank">George Brocklehurst</a> and the brilliant <a href="https://twitter.com/jcoglan" target="_blank">James Coglan</a>. James and I took a break to look at the <a href="http://www.flickr.com/photos/mn_francis/8559067275/in/set-72157632998241737/" target="_blank">causeway being underwater at high tide</a>, and when we returned we found that George had relocated from our previous office (the green sofa) to the flat roof of the officers' mess. With his phone playing Ben Howard's cover of <a href="http://www.youtube.com/watch?v=sPU8V-nvUEk" target="_blank">Call Me Maybe</a> and glasses of whisky hand-delivered by the fantastic <a href="https://twitter.com/cgovias" target="_blank">Chris Govias</a>, it was perfect.</p>
<p>Except it wasn't, because George's battery was running low, and he couldn't push his changes to my laptop because the wifi didn't reach that far. "Of course, we could always set up a local wifi on the roof," said George. Was he trolling me? No, he was not, and here's how:</p>
<p>George committed to a local branch and then set up his mac as a network hub (a <a href="http://support.apple.com/kb/PH3771" target="_blank">computer-to-computer network</a>).</p>
<p>Then I added a git remote that pointed at the repository on George's machine:</p>
<div class="code-sample">git remote add george georgebrocklehurst@169.254.145.66:code/devfort7/hobbit/.git</div>
<p>The IP address here was his machine on the network he'd created, and it was his ssh login (so I needed him to enter his password).</p>
<p>This included the pleasure of having to type:</p>
<div class="code-sample">git fetch george</div>
<p>Then I merged george's branch into mine:</p>
<div class="code-sample">git merge george/onboarding</div>
<p>and we were ready to use my computer to continue the good work.</p>
<img src="/img/roofhacking.jpg" />
<p class="photo-credit">Photo by the aforementioned brilliant <a href="https://twitter.com/jcoglan" target="_blank">James Coglan</a>.</p>
How to Estimate2013-01-21T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-to-estimate.html<p>I want to share this great idea about estimating that came from the excellent <a href="https://twitter.com/mnowster" target="_blank">Mazz Mosley</a>. Instead of worrying about estimating in hours or days, estimate in story points as follows:</p>
<div class="pretty-code-sample">
<p>1: This is trivial. I know exactly the code I would write if I went back to my desk right now.</p>
<p>2: This is quite easy. I know roughly what I'd have to do. I might have to look one or two things up.</p>
<p>3: This is a bit complex. I might have to refresh my memory on a few things and there are a couple of unknowns.</p>
<p>5: This is big. I have only a rough idea of how I'd do this.</p>
<p>8+: I don't know how to do this.</p>
</div>
<p>I've written 8+ rather than the more standard 8, 13, 21 etc because in our team we had an additional rule – if it's 8+ then the story is too big and you need to break it down a bit more. Maybe have a timeboxed spike? Then you will have more information for the next planning meeting.</p>
<p>It doesn't matter how much time each point ends up being (and this will vary from team to team); after a few sprints of estimating like this the velocity will become meaningful and you can use it for predicting how much work you'll get through in future sprints.</p>
Testing the redirections2012-12-10T00:00:00+00:00https://www.annashipman.co.uk/jfdi/testing-the-redirections.html<p>Now that <a href="https://www.gov.uk/">GOV.UK</a> has replaced Directgov and BusinessLink, and departments are
moving to <a href="http://digital.cabinetoffice.gov.uk/2012/11/15/a-quick-tour-of-inside-government/">Inside
Government</a>, we want to make sure that people visiting links to
the old sites get where they need to be. We want them to be redirected to the
correct page on GOV.UK, with <a href="https://gds.blog.gov.uk/2012/10/11/no-link-left-behind/">no link left
behind</a>.</p>
<p>This post is about the tools we built to make that possible.</p>
<h2 id="finding-the-links">Finding the links</h2>
<p>The first thing we needed was a list of everything we wanted to redirect: all
the Directgov and BusinessLink URLs (links and Web addresses). This proved to be
a fairly significant task - both sites had long histories and various different
providers, so a comprehensive list of these URLs did not exist.</p>
<p>Instead, we collected our own lists from a variety of sources, including
<a href="http://en.wikipedia.org/wiki/Web_traffic">traffic</a>
logs, records of friendly URLs (shorter, more memorable links that redirect to
longer URLs), and the results of
<a href="http://en.wikipedia.org/wiki/Web_crawler">spidering</a> the sites.</p>
<p>This gave us a total of about 8,000 Directgov URLs and about 40,000 BusinessLink
URLs.</p>
<h2 id="wrangling-the-urls">Wrangling the URLs</h2>
<p>Many of the lists of URLs existed in various spreadsheets, maintained by
different people. We needed a canonical source of truth. So we built the
<a href="https://github.com/alphagov/migratorator">Migratorator</a>.</p>
<p>The Migratorator is a <a href="http://rubyonrails.org/">Rails</a> app, backed by a
<a href="http://www.mongodb.org/">MongoDB</a> database. It allows
multiple users to create one-to-one mappings for each URL, where the mapping
consists of the source URL, status (whether it will be redirected or whether, no
longer representing a user need, it is now gone) and, if applicable, the page to
which it will be redirected.</p>
<p><img src="/img/migratorator_mapping.png" alt="Migratorator image" /></p>
<p>As well as the mapping information, the Migratorator allows us to capture other
useful information such as who has edited a mapping, tags showing information
about the type of mapping, and a status bar showing how far through the task we
are.</p>
<p><img src="/img/migratorator_filter.png" alt="Migratorator detail" /></p>
<h2 id="checking-the-mappings">Checking the mappings</h2>
<p>We needed to confirm that the mappings were actually correct. We wanted several
people to check each mapping, so we created the
<a href="https://github.com/alphagov/review-o-matic">Review-O-Matic</a>.</p>
<p>The Review-O-Matic is also a Rails app and uses the Migratorator API to display
the source URL and the mapped URL in a side-by-side browser, with voting
buttons.</p>
<p><img src="/img/review-o-matic.png" alt="Review-O-Matic" /></p>
<p>We asked everyone in GDS to help us by checking mappings when they had some
spare time. However, clicking through mappings can be dull, so we ran a
competition with a prominently displayed leader board. The winner, who checked
over 1,000 mappings, won cake.</p>
<h2 id="confirmation-from-departments">Confirmation from departments</h2>
<p>The Review-O-Matic presents the mappings in a random order, and the way it’s set
up means that links within pages cannot be clicked. This is good for getting as
many mappings as possible confirmed, but our colleagues in departments needed to
check content relevant to them in a more methodical and interactive way. Enter
the <a href="https://github.com/alphagov/review-o-matic-explore">Side-by-side Browser</a>.</p>
<p>The Side-by-side Browser displays the old and the new websites next to each
other. Clicking a link on the left hand side displays what this will redirect to
on the right hand side.</p>
<p><img src="/img/sidebyside.png" alt="Side-by-side browser image" /></p>
<p>The Side-by-side browser is a <a href="http://nodejs.org/">Node.js</a> proxy that serves itself and the site
being reviewed on the same domain, so that it’s ‘live’ and not blocked by the
<a href="http://www.w3.org/Security/wiki/Same_Origin_Policy">Same-Origin policy</a>. We joked that, in essence, the side-by-side browser was a
phishing attack for the good!</p>
<p>Initially it used the Migratorator API for the mappings. However, once we’d
built and deployed the Redirector, we could use that instead to populate the
right hand side. As well as simplifying the code, this meant we could now see
what the Redirector would actually return.</p>
<p>At this point, we distributed it to our colleagues in departments to check the
mappings and raise any concerns before the sites were switched over.</p>
<h2 id="bookmarklet">Bookmarklet</h2>
<p>We used another trick to test Directgov mappings while the site was still live.
We created a domain called aka.direct.gov.uk, which was handled by the
Redirector, and a <a href="http://whatfettle.com/2012/09/aka.html">bookmarklet</a>. By replacing the ‘www’ with ‘aka’ the bookmarklet
allowed us to see what an individual Directgov page would be replaced with.</p>
<h2 id="the-redirector-itself">The Redirector itself</h2>
<p>For the actual redirection, we use the open-source Web server
<a href="http://wiki.nginx.org/Main">Nginx</a>. The
<a href="https://github.com/alphagov/redirector/">Redirector project</a> is just the process for generating the Nginx configuration.
It’s written mainly in <a href="http://www.perl.org/">Perl</a> with some
<a href="http://php.net/">PHP</a>.</p>
<p>Generating the Nginx config requires logic to determine from the old URL what
kind of configuration should be used.</p>
<p>For example, the important part of a Directgov URL is the
<a href="http://en.wikipedia.org/wiki/URI_scheme#Generic_syntax">path</a>, e.g.
<a href="http://www.direct.gov.uk/en/Diol1/DoItOnline/Doitonlinemotoring/DG_196463">www.direct.gov.uk/en/Diol1/DoItOnline/Doitonlinemotoring/DG_196463</a>, while for
BusinessLink the essential information is contained in the query string, e.g
<a href="http://www.businesslink.gov.uk/bdotg/action/detail?itemId=1096992890&type=RESOURCES">http://www.businesslink.gov.uk/bdotg/action/detail?itemId=1096992890&type=RESOURCES</a>.
Redirecting these two types of URL requires different types of Nginx config.</p>
<p>This logic, plus the mappings we gathered, make up much of the Redirector
project.</p>
<h2 id="the-joy-of-tests">The joy of tests</h2>
<p>In addition, the project contains a suite of unit and integration tests,
including one that runs every night at 5am. This test checks that every single
URL in our source data returns a <a href="http://en.wikipedia.org/wiki/List_of_HTTP_status_codes">status
code</a> that is either a 410 ‘Gone’ or a
301 redirect to a 200 ‘OK’.</p>
<p>For a few weeks before the launch we also ran the daily Directgov and
BusinessLink logs against the Redirector to see if there were any valid URLs or
behaviour we’d missed. By doing this we found that, for example, even though
URLs are case-sensitive, Directgov URLs were not, and users would therefore
expect <a href="http://www.direct.gov.uk/sorn">www.direct.gov.uk/sorn</a> to work in the
same way as <a href="http://www.direct.gov.uk/SORN">www.direct.gov.uk/SORN</a>.</p>
<h2 id="going-live">Going live!</h2>
<p>The final task was to point the
<a href="http://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> for the sites we’re now hosting at the
Redirector. Now users following previously bookmarked links or links from old
printed publications will still end up on the right place on GOV.UK.</p>
<p><img src="/img/redirects.jpeg" alt="redirector post-it gets lots of votes at retrospective" /></p>
<p>The configuration now has over 83,000 URLs that we’ve saved from <a href="http://en.wikipedia.org/wiki/Link_rot">link
rot</a>, but
if you find an old BusinessLink or Directgov link that’s broken then let us
know.</p>
<p>Traffic through the Redirector is easing off as GOV.UK pages are consistently
higher in the Google search results, but it’s been really exciting making sure
that we do our best not to break the strands of the Web.</p>
<p><em>This post originally appeared on the <a href="https://gds.blog.gov.uk/2012/12/10/testing-the-redirections/">GDS Blog</a>.</em></p>
Learning the perls2012-11-20T00:00:00+00:00https://www.annashipman.co.uk/jfdi/learning-the-perls.html<p>After some time muddling through with Perl, I have accepted the inevitable – it's time to actually knuckle down and learn it properly. I have acquired <a href="http://www.amazon.co.uk/Learning-Perl-Randal-L-Schwartz/dp/1449303587/" target="_blank">some</a> <a href="http://www.amazon.co.uk/Modern-Perl-chromatic/dp/0977920178/" target="_blank">books</a>, but I also require some direction – so I asked my excellent colleague and friend <a href="https://twitter.com/cackhanded" target="_blank">Mark Norman Francis</a> to write a brief guest post for my blog.</p>
<p>His instructions were "Complete the sentence: In order to understand C you need to understand Pointer Arithmetic. In order to understand Perl you need to understand...". He went many better, and produced the following:</p>
<div class="pretty-code-sample">In order to write perl in a modern context, I think these are the basic skills you'll need.<br/>
<ol>
<li>Perl's provided documentation is extensive and worthwhile, so learn to use `perldoc` to read both the provided manuals and the documentation of the builtin perl functions.</li>
<li>Learn the difference between variables – scalar, array, hash, and references (hint for references: learn to make a hash of hashes).</li>
<li>Learn how to use an array as a stack.</li>
<li>Basic regular expression usage – matching, replacing, capturing substrings, case-insensitivity, metacharacters, and writing more readable expressions by having perl ignore the whitespace.</li>
<li>Learn to use perl on the command line as a pipe filter instead of awk/sed.</li>
<li>Learn how to write your own functions.</li>
<li>Learn how to use core modules such as `File::Find` (write a tool to list files in a directory structure from smallest to largest in size).</li>
<li>Refactor the code from step 7 to use a Schwartzian Transform to sort the file sizes.
<li>Learn how to install third-party libraries from CPAN.</li>
<li>Install Moose from CPAN, then work through `Moose::Manual` and `Moose::Cookbook` to learn to write modern OO perl.</li>
<li>Learn how to use `Test::More` to test code and `Devel::Cover` to produce coverage reports.</li>
<li>Find an abstract intro to OO example (such as employee payroll) and write it in Moose with full unit test coverage.</li>
<li>Lastly, read the `perlstyle` documentation, and then write your own style guide.</ol></div>
<p>Thanks, Norm. Thorm.</p>
Learning More About D32012-07-19T00:00:00+00:00https://www.annashipman.co.uk/jfdi/learning-more-about-d3.html<p>I had a really inspiring chat with <a href="https://twitter.com/mattb" target="_blank">Matt Biddulph</a> about D3 this week. He showed me some really cool things. I made some notes.</p>
<img src="/img/D3.jpg" />
CAP again – or should that be PACELC?2012-07-15T00:00:00+00:00https://www.annashipman.co.uk/jfdi/cap-again-or-pacelc.html<p>The last thing I wrote about was the CAP Theorem. Last week, <a href="https://twitter.com/thattommyhall" target="_blank">Tom Hall</a> visited our office and gave a very interesting talk, the central thesis of which was that the CAP Theorem, as often explained, is incorrect – or at least not very meaningful.</p>
<p>The first point, explained very clearly in <a href="http://www.cloudera.com/blog/2010/04/cap-confusion-problems-with-partition-tolerance/" target="_blank">CAP Confusion: Problems with 'partition tolerance'</a> is that P, "Partition Tolerance", as per the <a href="http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf"target="_blank">proof</a> of the CAP Theorem, is not a property of your application; rather it is a property of the network it's on. The network may partition. To quote from Henry Robinson's Cloudera article:</p>
<p class="pretty-code-sample">"Partition tolerance means simply developing a coping strategy by choosing which of the other system properties to drop. This is the real lesson of the CAP theorem – if you have a network that may drop messages, then you cannot have both availability and consistency, you must choose one."</p>
<p>The rest of the article is very worth reading.</p>
<p>The second point Tom covered is from <a href="http://dbmsmusings.blogspot.co.uk/2010/04/problems-with-cap-and-yahoos-little.html" target="_blank">Problems with CAP, and Yahoo's little known NoSQL system</a> by Daniel Abadi. Abadi covers the point above very clearly, and then points out that there is a missing letter from CAP – L, for latency.</p>
<p class="pretty-code-sample">"Keeping replicas consistent over a wide area network requires at least one message to be sent over the WAN in the critical path to perform the write... Unfortunately, a message over a WAN significantly increases the latency of a transaction... Consequently, in order to reduce latency, replication must be performed asynchronously. This reduces consistency."</p>
<p>He suggests CAP should be rewritten as PACELC:</p> <p class="pretty-code-sample">"If there is a partition (P) how does the system tradeoff between availability and consistency (A and C); else (E) when the system is running as normal in the absence of partitions, how does the system tradeoff between latency (L) and consistency (C)?"</p>
<p>Tom's talk covered many other interesting topics and is worth catching if you can. One of the other many interesting things Tom talked about was Vector Clocks. I won't go into that here, but they are worth reading up on! <a href="http://basho.com/blog/technical/2010/01/29/why-vector-clocks-are-easy/"target="_blank">Here</a> is a potential place to start...</p>
The CAP Theorem and MongoDB2012-04-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/the-cap-theorem-and-mongodb.html<p>This week I learned some things about MongoDB. One of them was about how it fits in with the CAP theorem.</p>
<p>They say a picture is worth a thousand words, and I think this diagram from my excellent new colleague <a href="https://twitter.com/matwall" target="_blank">Mat Wall</a> while he was explaining it to me says everything:</p>
<img src="/img/mongodb.jpg" />
<p>Over and out.</p>
<p>OK, perhaps I can offer a tiny bit of exposition.</p>
<p>The CAP Theorem is: where C is consistency, A is availability, and P is partition tolerance, you can't have a system that has all three. (It gets to be called a theorem because it has been formally proved.)</p>
<p>Roughly speaking:</p>
<ul>
<li>Consistency means that when two users access the system at the same time they should see the same data.</li>
<li>Availability means up 24/7 and responds in a reasonable time.</li>
<li>Partition Tolerance means if part of the system fails, it is possible for the system as a whole to continue functioning.</li>
</ul>
<p>If you have a web app backed by a SQL database, most likely, it is CA.</p>
<p>It is C because it's transaction-based. So when you update the database, everything stops until you've finished. So anything reading from the database will get the same data.</p>
<p>It can be A, but it won't be P because SQL databases tend to run on single nodes.</p>
<p>If you want your application to be P, according to the CAP theorem, you have to sacrifice either A or C.</p>
<p>With MongoDB, in order to gain P, you sacrifice C. There are various ways to set it up, but in our application we have one master database, that all writes go to, and several secondaries (as can be seen from the diagram: M is the Master, the Rs are the secondaries – also called replicas, or slaves). Reads may come from the secondaries. So it is possibly that one or more of the secondary nodes could be disconnected from the application by some kind of network failure, but the application will not fall over because the read requests will just go to another node. Hence P.</p>
<p>The reason this sacrifices C is because the writes go to the master, and then take some time to filter out to all the secondaries. So C is not completely sacrificed – there is just a possibility that there may be some delay. We are not allowing a situation where the secondaries are permanently out of synch with the master – there is "eventual consistency". </p>
<p>So you might use this in applications where, for example, you are offering the latest news story. If User A gets the latest news 10 seconds earlier than User B, this doesn't really matter. Of course, if it was a day later, then that would be a problem. The failure case of C is just around the time of the write and you want to keep that window of consistency small.</p>
<p>There is also a concept of durability, which you can also be flexible with.</p>
<p>Take the following two lines of pseudocode:</p>
<p class="code-sample">
1. insert into table UNIVERSAL_TRUTHS (name, characteristic) values ('Anna', 'is awesome')<br/>
2. select characteristic from UNIVERSAL_TRUTHS where name = 'Anna'
</p>
<p>What we're saying when we sacrifice consistency is, if I run these two lines on the same node then when I run line 2, I can be sure it will return 'is awesome'. However, if I run line 2 on a different node, I can't be sure it's in already. It will still be "eventually consistent" so if I run it later (and it hasn't been changed again in the interim) it will at some point return the correct data.</p>
<p>However, you can also configure MongoDB to be flexible about durability. This is where, if you run the two lines of code on the <date>same</date> node, it might be the case that line 2 hasn't run, and possibly even never will. You might do this, for example if you were storing analytics. If you are looking for general trends, it might not matter so much if 1% of the transactions fail, so you might configure it to be flexible on durability. Of course you wouldn't do that for something as crucial as characteristics about Anna.</p>
What Should Your Work Log Tell You?2012-04-15T00:00:00+00:00https://www.annashipman.co.uk/jfdi/what-should-your-work-log-tell-you.html<p>We've got some great data for the <a href="http://www.spaconference.org/spa2012/" target="_blank">SPA2012</a> workshop on JavaScript visualisations. <a href="https://twitter.com/hibri" target="_blank">Hibri</a> has managed to obtained three years' worth of <a href="http://www.7digital.com/" target="_blank">7Digital</a>'s work-tracking data and <a href="https://twitter.com/robbowley" target="_blank">Rob</a> has very kindly allowed us to use it (anonymised, of course).</p>
<p>We're going to use this dataset to show different information using different libraries for the visualisations.</p>
<p>What I'd really like your input on is, what would you like to know? If this was your work-tracking data, what questions would you want it to answer?</p>
<p>For example, we could create a Sunburst diagram using JIT to show what proportion of total development time is spent on the four different applications. We could create an interactive line graph using Raphael to show the average time a piece of work takes, and how that average has changed over the past three years.</p>
<p>A little more info about the exact data is below, or feel free to skip straight to the <a href="#example-questions">example questions</a>, or just go right ahead and <a href="https://twitter.com/annashipman" target="_blank">tweet me</a> or email me (at this domain) with what kind of information you'd want from a work log stretching back that far. You might get a great visualisation prepared for you to show just that!</p>
<h2>The Data</h2>
<p>Here is an example:</p>
<p class="code-sample">
"Feature ID": "A-321",</br>
"Application Name": "Project A",</br>
"Type": "Bug",</br>
"T-Shirt Size": "S",</br>
"Added": "15/03/2011 12:12",</br>
"Prioritised, Awaiting Analysis": "15/03/2011 12:12",</br>
"Analysis Completed": "15/03/2011 12:12",</br>
"Development Started": "15/03/2011 13:21",</br>
"Systest Ready": "15/03/2011 16:21",</br>
"Systest OK": "17/03/2011 10:34",</br>
"In UAT": "17/03/2011 10:34",</br>
"Ready For Release": "17/03/2011 10:34",</br>
"In Production": "17/03/2011 10:35",</br>
"Done": "22/03/2011 10:48",</br>
"Lead Time": "2",</br>
"Cycle Time": "2",</br>
"Development Cycle Time": "2"
</p>
<p>So this is a really rich dataset. We have all the dates that various stages of the release process happened, as well as some information about what application it was for, what type of work it was (e.g. Bug, Feature, Build Maintenance), what the rough estimate was (T-shirt size), and how long the whole process took, both from first raised to done (lead time) and from dev start to done (development cycle time). The log goes back to 2009.</p>
<h2 id="example-questions">Example Questions</h2>
<p>Here are a few of the questions we've thought about answering. </p>
<ul><li>How many bugs are raised by application?</li>
<li>How does the average cycle time of development compare to the number of bugs raised?</li>
<li>How often are bugs reported and not worked on?</li>
<li>What proportion of time is spent on each applications, and, drilling down, what proportion of that work is new features/bugs/build maintenance etc?</li></ul>
<h2>Your Chance for a Hand-crafted Visualisation Just for You!</h2>
<p>We'd love this workshop to be as relevant and interesting as possible, so please do let me know what kind of information you'd like to see visualised and it may well make it into the talk! <a href="https://twitter.com/annashipman" target="_blank">Tweet me</a> or email me at this domain. Thank you!</p>
Plan of Action2012-02-20T00:00:00+00:00https://www.annashipman.co.uk/jfdi/plan-of-action.html<p>My goal is to be a better developer.</p>
<p>A few weeks ago, I had an appraisal. There was a lot of positive feedback and
also some really useful suggested areas for development. With the help of my
good friend and unofficial mentor <a href="http://www.twitter.com/boffhaku" target="_blank">Dolan O'Toole</a>, I pulled together some practical suggestions as to how I could improve them:</p>
<img src="/img/dolsfeedback.jpg" />
<p>I then drew up a rough syllabus of the next steps I should focus on:</p>
<img src="/img/syllabus.jpg" />
<p>I'd be interested if any of you have suggestions for additions to this? What should a great developer know?</p>
Would Have Been Useful to Know at My First Game Jam2012-02-14T00:00:00+00:00https://www.annashipman.co.uk/jfdi/would-have-been-useful-to-know-at-my-first-game-jam.html<p>I had no idea what to expect at my first <a href="http://globalgamejam.org/" target="_blank">game jam</a>, and I was nervous about whether I'd be able to participate. I'm a programmer, but I'd never written a videogame, and I don't even much play videogames.</p>
<p>As it turned out, everyone was really friendly and welcoming and I was able to get totally stuck in. But there were a few things that I didn't know that everyone else seemed to know, and I was too shy to admit I didn't know too. Now, following my new and brilliant strategy of "ask as many questions as you can, the stupider the better" (more on that in a later post), I have managed to learn some of these things, and I'll share them here. They are <a href="#game-engine">the game engine</a>, <a href="#game-mechanics">game mechanics</a>, and <a href="#art">a little bit about art</a>.</p>
<h2 id="game-engine">The Game Engine</h2>
<p>When writing video games, people talk about the "engine". For example, "What engine are you using?" and "Don't write your own engine; this is not 'engine jam' ".</p>
<p>A game engine is something that handles the nuts and bolts of a game for you, which basically comes down to displaying the graphics, dealing with input (keyobard, joystick etc), and handling collisions. For example, in a game, when your player runs into a baddie you might want them to bounce off each other at an equal pace. Using a game engine means you can just specify the details of the collision. Not using a game engine means you have to write code to describe the angles and velocities yourself.</p>
<p><a href="http://unity3d.com/unity/" target="_blank">Unity</a> is a game engine that handles all this stuff for you. You can copy your art into the folder structure and then drag and drop it to where you want it to be. Game objects have default collision behaviour and you can just write a script to change it.</p>
<p><a href="http://www.flixel.org/" target="_blank">Flixel</a> is another game engine. It's a level below Unity – there's no graphical user interface, and you have to do a bit more work with the graphics, but the collisions, keyboard input and various other features are there for you.</p>
<h3>A Note on Writing Your Own Game Engine</h3>
<p>I didn't understand at the time why my <a href="http://globalgamejam.org/2011/flocks-sake" target="_blank">second game jam</a> was so much harder than my <a href="http://globalgamejam.org/2010/bacterial-warfare" target="_blank">first</a>, but when I explained what we did to the brilliant <a href="https://twitter.com/joehalliwell" target="_blank">Joe Halliwell</a> afterwards, it all became clear. We disobeyed the advice. We wrote our own game engine.</p>
<p>To be precise, we used <a href="http://www.opengl.org/about/" target="_blank">OpenGL</a>, which is a 2D/3D Graphics API written in C. It gives the instructions to the graphics card – you definitely don't want to be doing that yourself – at least not in a 48-hour game jam. The code was written in Python, and for the binding we used <a href="http://pyopengl.sourceforge.net/" target="_blank">PyOpenGL</a>. For the physics we used <a href="http://box2d.org/" target="_blank">Box2D</a>. OK, so we didn't exactly write our own game engine from scratch, but we assembled one from various constituent parts.</p>
<p>The alleged reasons were that we didn't know Unity, most of us knew Python (except for me – that added another layer of complexity to the weekend!) and between us we had two Macs, two Linux boxes and one loser running Windows. The real reason is that the brilliant <a href="http://www.texasexpat.net/" target="_blank">Trevor Fountain</a> loves this stuff and, if it existed, would sign up to 'engine jam' like a shot.</p>
<h2 id="game-mechanics">Game Mechanics</h2>
<p>Another phrase I heard a lot and pretended to understand was "game mechanics". To be fair, I thought I did understand it – I thought it was jumping, shooting etc. I was wrong.</p>
<p>Game mechanics are more like constructs of rules. Unlocking an achievement is a game mechanic. Going up a level when you reach a certain number of points is a game mechanic. Dying when you lose 90% of your health bar is a game mechanic. So when someone asks "What are the game mechanics?", they mean something like, what are the gamey things here? What makes this a game, rather than just graphics moving across a screen?</p>
<p>I like <a href="http://gamification.org/wiki/Game_Mechanics" target="_blank">this list</a> of potential game mechanics.</p>
<h2 id="art">A Little Bit About Art</h2>
<p>While I'm here, just a few words on pixel art vs vector graphics, something else I heard talked about which added to the background radiation of feeling I didn't quite understand what was going on.</p>
<p>There's an extremely detailed article on the differences <a href="http://www.1stwebdesigner.com/design/pixel-vector-graphics-difference/" target="_blank">here</a> but in very brief summary – technically speaking, pixel (aka bitmap, raster) art is basically dot-matrix, i.e. made up of pixels individually coloured, and vector art is created using mathematical functions and is basically points and the lines joining them. The art for your game may be either, or some combination of the two. A brief rule of thumb is that old games tend to use pixel art, but the choice is up to you – different tools for different times.</p>
<p>However, it's more than just the technical differences, and the conversations you hear about pixel vs vector art will probably reflect something other than the strict technical distinction. <a href="https://twitter.com/doches" target="_blank">Trevor</a> summed it up for me very clearly, so here is an edited version of what he said:</p>
<p>"To me, pixel art is more of an aesthetic choice than a technical term. It's true that older games had a pixel art aesthetic, especially on 8- or
16-bit platforms. Today, though, I think people use the terms 'pixel art' and 'retro' more or less interchangeably – games like <a href="http://www.canabalt.com" target="_blank">Canabalt</a> or <a href="http://shauninman.com/lastrocket/" target="_blank">The Last Rocket</a> are undeniably retro, and use the pixel art style to great effect. Games like <a href="http://chrome.angrybirds.com/" target="_blank">Angry Birds</a> are rendered using the same technology (2d sprites), yet no one would ever say that Angry Birds qualified as pixel art.</p>
<p>"As for vector art, true vector images (e.g. svg) are rarely used outwith UI design and Flash."</p>
<h2>Next steps</h2>
<p><a href="http://v21.posterous.com/how-to-make-your-first-videogame" target="_blank">How to Make Your First Videogame</a> by the excellent <a href="https://twitter.com/v21" target="_blank">v21</a> is great, and the short video <a href="http://youcanmakevideogames.com/" target="_blank">You Can Make Video Games</a> is fab and really worth watching. Or why not just sign up to the <a href="http://globalgamejam.org/" target="_blank">next Global Game Jam</a>?</p>
<p>Enjoy!</p>
JavaScript Talk Takes Shape2012-01-21T00:00:00+00:00https://www.annashipman.co.uk/jfdi/javascript-talk-takes-shape.html<p>Just over a week to go until the final deadline for the SPA submissions, and the JavaScript visualisations talk is shaping up.</p>
<p>The plan is to demonstrate the use of a JS library to produce a certain kind of visualisation, followed by a 15-20 minute exercise where participants use the library to create a similar visualisation. I've said "rinse and repeat 4-5 times", though I'm slightly dubious as to whether we're going to have time to do more than three. <a href="https://twitter.com/hibri">Hibri</a> and I are going to do one each and demo to the other, so we'll have a better idea then of how many we can cover. I'm really excited about what we're going to produce!</p>
<p>At the moment we are planning to try and do:</p>
<ul>
<li>A force-directed graph (library tbc, possibly d3)</li>
<li>A sunburst diagram with JIT</li>
<li>An interactive line graph with Raphael</li>
<li>A bar chart (library tbc, possibly just using Canvas)</li>
<li>A stream graph (library tbc, possibly d3 if not using d3 for force-directed)</li>
<li>A heat map (library tbc)</li>
</ul>
<p>Is there a type of visualisation you'd really like to see us demonstrate, or a library you think we can't miss out? Please do let me know.</p>
<p>There is also still time to give feedback directly on this and all other proposals for SPA 2012 <a href="http://www.spaconference.org/scripts/proposals.php">here</a> (SPA login required).</p>
Preparation Begins for the JavaScript Talk...2011-12-18T00:00:00+00:00https://www.annashipman.co.uk/jfdi/preparation-begins-for-javascript-talk.html<p>As part of preparing the proposal for the JavaScript visualisations talk, I had a coffee with <a href="https://twitter.com/ian_alderson" target="_blank">Ian Alderson</a>. He did an excellent talk at SPA2011, along with Adam Iley, on HTML5. The presentation is still available online (using HTML5 for the slides!) <a href="http://webappsthatshine.com/presentation/" target="_blank">here</a>, I really recommend you check it out.</p>
<p>Although Ian doesn't have time to co-pilot on the talk, he very generously gave up his time to give me some advice. Notes below. The first suggestion of his that I plan to follow is a comparison between SVG and Canvas – as he pointed out, all JS visualisation libraries will be based on one or the other, so that will definitely be a factor in what library to choose. It's not the case that one is just better than the other – both have strengths and weaknesses. Watch this space for more on this topic...</p>
<img src="/img/talkwithIan.jpg" />
JavaScript Visualisations Proposal2011-11-30T00:00:00+00:00https://www.annashipman.co.uk/jfdi/javascript-visualisations-proposal.html<p>I have proposed a talk for SPA2012 on JavaScript visualisations.</p>
<p>The basic outline of the proposal is a workshop where we will demonstrate the use of different JS libraries to visualise different kinds of data with several 15-20 minute exercises where participants use the library to create a similar visualisation. Ideally participants would leave with an overview of how and when to use the libraries and tools we've covered, and ideally an idea of what factors to think about when choosing a JS data visualisation tool.</p>
<p>Still in the very early stages of planning it, but I am in the process of moving this site over to a host that will support WordPress, at which point I will be able to invite comments/suggestions. In the meantime, any suggestions are very welcome, by email or <a href="http://www.twitter.com/annashipman" target="_blank">twitter</a>.</p>
<p>Notes from the first planning meeting, with <a href="http://www.hibri.net/" target="_blank">Hibri</a>:</p>
<img src="/img/planning1.jpg" />
My Continuing Struggles with Linux2011-10-17T00:00:00+00:00https://www.annashipman.co.uk/jfdi/my-continuing-struggles-with-linux.html<p>Summarised excellently by this cartoon. It's great fun, but...</p>
<p><a href="http://xkcd.com/196/" target="_blank"><img src="/img/command_line_fu.png" /></a></p>
Sorl-thumbnail Comes Back to Bite Me2011-09-12T00:00:00+00:00https://www.annashipman.co.uk/jfdi/sorl-thumbnail-comes-back-to-bite-me.html<p>Well, the <a href="#no-sorl-thumbnail">lack of sorl-thumbnail</a> came back to bite me when I got the latest version of the Django project I've been working on. Previous releases had lulled me into a false sense of security, but then the latest update, a bit too late in the evening, I had that old error:</p>
<p class="code-sample">ImportError: No module named sorl-thumbnail</p>
<p>So off I went again. I had no problem downloading sorl-thumbnail but everywhere I went wanted to put it into the wrong directory. It's not in the Synaptic Package Manager, and I couldn't find a PPA. Just to make it a bit more exciting, the dependency of the project I needed it for was something like version 3.2.5. The current version is 11.0.4.</p>
<p>Ideally what I wanted to be able to do was set some kind of flag on the install that told it to install somewhere else. Surelly there is a way to do that? But I couldn't find it. I had got as far as actually (again!) RTFM, and was browsing through the <a href="http://docs.python.org/distutils/apiref.html" target="_blank">Core Distutils functionality</a> page when lo and behold, I stumbled upon <a href="http://uswaretech.com/blog/2009/03/create-your-own-online-store-in-few-hours-using-satchmo-django/" target="_blank">this</a>. Not directly relevant you might think – but wait, item 13? What's this?</p>
<p>Yes, to cut a long story short, I had discovered symlinks. New to me, old to everyone else. It was short work to set one up, helped by <a href="http://www.cyberciti.biz/faq/creating-soft-link-or-symbolic-link/" target="_blank">this</a>, and lo and behold: it works.</p>
<p>Kind of begs the question as to why I didn't do that in the first place for all the packages... but never mind. Onwards and upwards!</p>
How to Write Efficient CSS – Evaluation of CSS2011-08-29T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-to-write-efficient-css-evaluation-of-css.html<p>I recently did a knowledge share at work on how to write efficient CSS, and one of my colleagues – a senior developer with a lot of experience – said "I felt the penny drop". High praise indeed, so I thought it was worth reproducing it here.</p>
<p>The write-up is pretty long, so I've divided the divs into <a href="#basic-css">Basic CSS</a> in which I cover:
<ol>
<li><a href="#css">CSS</a></li>
<li><a href="#selectors">Selectors</a></li>
<li><a href="#the-cascade">The Cascade</a></li>
<li><a href="#inheritance">Inheritance</a></li>
</ol>
and <a href="#evaluation-of-css">Evaluation of CSS</a></li> in which I cover how CSS is evaluated and some things we can do to make it more efficient.</p>
<h2>How CSS is Evaluated</h2>
<p>The key information is these two points:
<ol>
<li>For each element, the CSS engine searches through style rules to find a match.</li>
<li>The engine evaluates each rule from right to left, starting from the rightmost selector (called the "key") and moving through each selector until it finds a match or discards the rule.</li>
</ol>
<h3>1. For each element, the CSS engine searches through style rules to find a match</h3>
<p>The style system breaks rules up into four categories by key selector.
<ol>
<li>ID Rules – for example <span class="pretty-code-sample">button#backButton { ... }</span></li>
<li>Class Rules – for example <span class="pretty-code-sample">button.toolbarButton { ... } </span></li>
<li>Tag Rules – for example <span class="pretty-code-sample">treeitem > treerow { ... } </span></li>
<li>Universal Rules – all other rules</li>
</ol>
<p>The CSS engine then takes each html element in the document in turn. If it has an ID, then the engine searches through the style rules and checks rules that match that element's ID. If it has a class, only Class Rules for a class found on the element will be checked. Only Tag Rules that match the tag will be checked.
Universal Rules will always be checked.</p>
<h3>2. The engine evaluates each rule from right to left...</h3>
<p>So in the example <span class="pretty-code-sample">button#backButton { ... }</span> the key is the id "backButton". In the example I give in <a href="#basic-css">Basic CSS</a> the key is the class "blog-post".</p>
<p>The engine starts with the key and then evaluates the rule from right to left. So if you have a button with an id of "backButton", the engine first matches the id to the id and then compares the next selector – is the element a button? In the example from <a href="#basic-css">Basic CSS</a> the evaluation for the second selector, <span class="pretty-code-sample">ul#nav.dhtml li a.blog-post</span>, is as follows. Does the element have a class of blog-post? If so, is it a link? If so, is there anywhere in its ancestry a list item? If so, is there anywhere in the ancestry of that list item an unordered list element with a class of dhtml and an id of nav?</p>
</p>You may be getting a slight clue now as to why I think that selector is inefficient.</p>
<h3>Recommendations</h3>
<p>There are some obvious ways we can start here. </p>
<ul><li>Firstly, remove unused CSS. It's not like, for example, unused Java code which really just slows things down for other developers. The CSS engine is going to search through every rule for every element. The fewer rules required to check for a given element, the faster style resolution will be.</li>
<li>Avoid use of descendant selectors – they speed up page development, but they slow down page rendering.</li>
<li>Avoid qualifying ID selectors at all – as IDs should be unique so adding extra qualification is unnecessary and costly.</li>
<li>It's a good idea to avoid use of html elements in selectors for a couple of reasons:
<ol>
<li>The style will be lost of the element type is changed</li>
<li>The less specific the key the greater the number of nodes that need to be evaluated</li>
</ol>
<li>Avoid use of child selectors. They are more efficient than descendant selectors, but they are still inefficient: for each matching element, the browser has to evaluate another node. It becomes doubly expensive for each child selector in the rule.</li>
</ul>
<p>These are just recommendations as to how to write the CSS. If you care about your website's performance you should already be <a href="http://developer.yahoo.com/performance/rules.html" target="_blank">minimising and gzipping your CSS</a>. So how much of an edge will these recommendations give you?</p>
<h3>CSS and Performance</h3>
<p>CSS and performance is a fairly hot topic right now, especially with all the cool things that you can do using CSS3. Dave Hyatt, architect for Safari and Webkit, said "The sad truth about CSS3 selectors is that they really shouldn't be used at all if you care about page performance." (The comment can be found <a href="http://www.shauninman.com/archive/2008/05/05/css_qualified_selectors#comment_3942" target="_blank">here</a>).</p>
<p>That's certainly something I've heard, for example at conferences. However, another web giant, Steve Souders (works at Google on web performance) has a different opinion. It's worth reading the piece <a href="http://www.stevesouders.com/blog/2009/03/10/performance-impact-of-css-selectors/" target="_blank">in full</a> (there are charts and everything!), but the takeaway here is: "On further investigation, I'm not so sure that it's worth the time to make CSS selectors more efficient. I'll go even farther and say I don't think anyone would notice if we woke up tomorrow and every web page's CSS selectors were magically optimized."</p>
<p>So why am I bothering with this? Well, a few reasons. One is I think it's always worth taking the time to find out how things work and I'm glad to be able to make a reasoned judgment on this.</p>
<p>But also, there are various things to consider when thinking about writing efficient CSS. There is performance, obviously, but two other major concerns are ease of writing and (in my view more important) ease of reading. I love clean code, and I think it's crucial that code is easy for other developers to read. I'm not sure I care whether or not <span class="pretty-code-sample">ul#nav.dhtml li a.blog-post</span> is performant or not, it's certainly not clean, and it took me some brow furrowing to work out when it would apply. So personally, I'm going to follow the rules I've outlined above. What do you think? I'd love to hear your opinion.</p>
<h3>Useful Further Reading</h3>
<p>I gleaned most of the information about how the CSS engine works from <a href="http://code.google.com/speed/page-speed/docs/rendering.html" target="_blank">Page Speed documentation</a> and the <a href="https://developer.mozilla.org/en/writing_efficient_css" target="_blank">Mozilla blog</a>. I couldn't find any information about how IE evaluates CSS, please do let me know if you have any.</p>
<p>To take things further with writing nice CSS, you could look at <a href="http://net.tutsplus.com/tutorials/html-css-techniques/using-compass-and-sass-for-css-in-your-next-project/" target="_blank">Compass/SASS</a>, or my current favourite, <a href="http://oocss.org/" target="_blank">OOCSS</a>.</p>
How to Write Efficient CSS – Basic CSS2011-08-06T00:00:00+00:00https://www.annashipman.co.uk/jfdi/how-to-write-efficient-css-basic-css.html<p>I recently did a knowledge share at work on how to write efficient CSS, and one of my colleagues – a senior developer with a lot of experience – said "I felt the penny drop". High praise indeed, so I thought it was worth reproducing it here, even though other similar divs are available on the internet.</p>
<p>The write-up is pretty long, so I've divided the divs into <a href="#basic-css">Basic CSS</a> in which I cover:
<ol>
<li><a href="#css">CSS</a></li>
<li><a href="#selectors">Selectors</a></li>
<li><a href="#the-cascade">The Cascade</a></li>
<li><a href="#inheritance">Inheritance</a></li>
</ol>
and <a href="#evaluation-of-css">Evaluation of CSS</a></li> in which I cover how CSS is evaluated and some things we can do to make it more efficient.</p>
<h2 id="css">CSS</h2>
<p>For those who have no idea at all what CSS is, I can recommend nothing better than the <a href="http://www.w3schools.com/css/css_intro.asp">w3schools excellent intro</a>. The w3school is always my first port of call for definitive answers on html, CSS and JavaScript. I will assume that you at least know that CSS stands for Cascading Style Sheets, and realise why separating that from html can make development easier.</p>
<p>
<h3>Syntax</h3>
<p>CSS is applied to html elements using selectors. The syntax of CSS is: </p>
<span class="pretty-code-sample">selector { CSS property : value; }</span>
<p>If you want to add comments, use <span class="pretty-code-sample">/* comment */
</span> as C-style comments (//) will cause your CSS to fail silently – all you'll see is your website looking wrong.</p>
<p>Let's look at some more details of how CSS is written. </p>
<p class="pretty-code-sample">a.blog-post, ul#nav.dhtml li a.blog-post {<br/>
<span>display: block;</span><br/>
<span>float: right;</span><br/>
<span>height: 24px;</span><br/>
<span>width: 24px;</span><br/>
<span>margin: 3px 3px 0 0;</span><br/>
<span>padding: 0 0 0 0;</span><br/>
<span>background: url(../images/film_help.png) no-repeat;</span><br/>
}</p>
<ol>
<li><h3>Padding and the box model</h3></li>
<p>If you've ever worked with CSS at all, no doubt you have fiddled around changing the padding or the margin without actually knowing why. I certainly did. Until I saw this diagram, which suddenly made everything clear:</p>
<a href="http://www.w3schools.com/css/css_boxmodel.asp" target="_blank"><img src="/img/box-model.gif" /></a>
<p>Have a look at the page on w3schools, but basically the margin is how far the box is from other elements, and the padding is how much space you'd like between the inside of the box and the content. Genius. And obviously, although not given in the CSS example above, you can manipulate the border too, as I have done in my code samples on this page, for example <span class="code-sample">border-style: dashed;</span></p>
<li><h3>Padding and Margin shortcuts</h3></li>
<p>It is possible to specify each direction of the padding, for example <span class="pretty-code-sample">padding-left:3px</span>. However you can also use a shortcut, as has been done in the example. The order is like a clock: Top, Right, Bottom, Left; so an element with the CSS in the example applied to it will have a margin on the top and the right. <a href="http://stackoverflow.com/questions/356759/a-mnemonic-for-the-order-of-css-margin-and-padding-shorthand-properties" target="_blank">Another way to remember this</a> is by remembering that if you don't get it right there will be TRouBLe. :o)</p>
<p>However, you can make it even shorter than that. Three values set the top to the first, the right AND left to the second and the bottom to the third. For example:</p>
<p>margin:10px 5px 15px;<br/>
top margin is 10px<br/>
right and left margins are 5px<br/>
bottom margin is 15px<br/>
</p>
<p>Two values set the top and bottom to the first, and the right and left to the second, and one value sets them all to the same. <a href="http://www.w3schools.com/css/css_margin.asp" target="_blank">Here</a> if you didn't follow that.</p>
<li><h3>Ems</h3></li>
<p>Sizes can be given in pixels, ems or percentages. Pixels you know. Percentages are as a percentage of the enclosing elements (see <a href="#inheritance">inheritance</a>). Ems are also calculated as percentages, where the default for most browsers is that 1em = 16px. However (of course) there is a slight twist in that ems are not calculated correctly in IE, so .85em is smaller than 85%. The <a href="http://www.w3schools.com/css/css_font.asp" target="_blank">w3schools recommendation</a> is to use a combination of percentages and ems. The <a href="#developer-tools">developer tools</a> are invaluable in this respect and there is a useful discussion <a href="http://www.jaxbeachtech.com/css-font-size" target="_blank">here</a>.</p>
<p>In the meantime you might want to remember this hilarious joke: <date>Q. What did one em say to the other em? A. Who's your daddy?</date> </p>
<li><h3>No-repeat</h3></li>
<p>A brief word on the image. <span class="pretty-code-sample">background: url(../images/film_help.png) no-repeat;</span>. The url is a link to where you have stored the image. <span class="pretty-code-sample">no-repeat</span> means you want it to appear once. The default is for it to repeat both horizontally and vertically. A nice trick for filling in a background is to create a slim image with the gradiations you want and then <span class="pretty-code-sample">repeat-x</span> so it fills the div.</p>
<p>An aside: another way to include images is to instead use a data url. This is where you encode the image inline. The advantage of this is that it doesn't require an extra HTTP request to get the image, and the key to a faster website is minimising the HTTP requests. However, there are reasons (other than the ubiquitous incomplete cross-browser support) why you might not want to use data urls – a good discussion can be found <a href="http://stackoverflow.com/questions/1124149/embedding-background-image-data-into-css-as-base64-good-or-bad-practice" target="_blank">here</a>. Other ways to minimise the HTTP requests associated with loading images can be found <a href="http://developer.yahoo.com/performance/rules.html" target="_blank">here</a>.</p>
<li><h3>Optional;</h3></li>
<p>The last semi-colon is optional. In fact, the structure of a CSS class is <span class="pretty-code-sample">selector { CSS property : value }</span> – you only need the semi-colon if you have subsequent instructions. I always put it in anyway though, there's no reason not to.</p>
</ol>
<h2 id="selectors">Selectors</h2>
<p>There are <a href="http://www.w3schools.com/cssref/css_selectors.asp" target="_blank">loads</a> of selectors in CSS. For now, I'll just talk about the most common:</p>
<h3>html element</h3>
For example, <span class="pretty-code-example">p { color: red; }</span> would make all text in paragraphs red.
<h3>#id</h3>
Giving the ID makes the one specific element behave in a certain way. It is certainly the most efficient selector (see <a href="#evaluation-of-css">how it's evaluated</a>, but it isalso slightly pointless. The point of style is to make a website look and feel consistent – with your brand, with itself, etc. IDs are (supposed to be) unique on a page. So why would you want to style just one thing?
<h3>.class</h3>
The most common selector and the one I think we should all be using most of the time.
<h3>*</h3>
The universal selector. Some interesting discussion <a href="http://meyerweb.com/eric/divs/webrev/200006a.html" target="_blank">here</a>, but in summary it is the least efficient selector and also allows you to bypass inheritance. I say AVOID.
<h3>Combinations</h3>
<h4>x, y</h4>
<p>This means, apply the style you are outlining to all xs and all ys. For example <span class="pretty-code-sample">p, .blog-intro { font-weight: bold; }</span> makes the text in all paragraphs bold AND the text in all elements with the class "blog-intro".</p>
<h4>x y</h4>
<p>The descendant selector. This matches any y which is a descendant of any x. For example <span class="pretty-code-example">p .blog-intro { font-weight: bold; }</span> makes all the text within a paragraph that has the class of "blog-intro" bold – even if it is nested several layers within. It is very inefficient. See <a href="evaluation-of-css">later post</a></p>
<h4>x > y</h4>
<p>The child selector. This matches any y which is a child of any x. This is also inefficient, though less inefficient than the descendant selector.</p>
<h3>a.blog-post, ul#nav.dhtml li a.blog-post</h3>
<p>So, to return to the selector in the example which I have so far ignored.</p>
<p>To spell it out, the style outlined here will be applied to any link with the class of blog-post, AND any link with the class of blog-post which is a descendant of a list item which is itself a descendant of an unordered list with the id of nav and a class of dhtml.</p>
<p>Pretty complicated. Over-complicated, I will argue. But just to note – the reason you might want both of these (you might think the first would cover the second) is because you may well have one or more intermediate styles that have altered the link in the list item further. For example if you have styled ul#nav.dhtml differently, the link will appear differently unless you reset it here.</p>
<p>In my view, examples like this are why people do not like CSS. Oh, and this is a real example from a live application, by the way. Names slightly changed to protect the innocent.</p>
<h2 id="the-cascade">The Cascade</h2>
<p>As hinted at above, you can have multiple styles applied to an element, and the cascade is what determines which one is actually applied. CSS styles are cascaded in the following order, with the one applying last being the one that sticks:
<ol>
<li>Origin (i.e. Browser then Author)</li>
<li>Importance (normal vs !important)</li>
<li>Specificity
<ol>
<li>Inline</li>
<li>Highest number of id selectors</li>
<li>Highest number of class, attribute, or pseudo classes</li>
<li>Highest number of elements and pseudo elements</li>
</ol></li>
<li>If still the same, the order in which they appear.</li>
</ol>
<p><a href="http://reference.sitepoint.com/css/cascade" target="_blank">Here</a> for (very dense) more.
<h3>Specificity</h3>
<p>Selectors are considered to be specific in the following way (ordered from the least to the most specific):
<ol>
<li>The type selector</li>
<li>The descendant selector</li>
<li>The adjacent sibling selector</li>
<li>The child selector</li>
<li>The class selector</li>
<li>The attribute selector</li>
<li>The ID selector</li>
</ol>
<a href="http://www.alternategateways.com/tutorials/css/css-101/part-four-the-css-order-of-precedence" target="_blank">Here</a> for more.
<h3>Order</h3>
<p>The order, listed from the least precedence to the most precedence:</p>
<ol>
<li>Browser default</li>
<li>External style sheet</li>
<li>Internal style sheet (i.e. written in the head)</li>
<li>Inline style (i.e. inside an HTML element)</li>
</ol>
<p>Notes:</p>
<p>An external stylesheet will override an internal stylesheet if it's linked to in the head after the internal style sheet.</p>
<p>And the most common trip-up in legacy code systems: The same class within a CSS file will override one written higher up in that file.</p>
<h3>important!</h3>
<p>You will have seen that <span class="pretty-code-sample">important!</span> takes precedence in the cascade. This is what you add if you can't get the style you want using the normal rules of cascade and inheritance. Don't use it unless you <strong>really</strong> have to – to me, it says "I don't know how CSS works so I'm just going to hack it."</p>
<h3 id="developer-tools">Developer Tools</h3>
<p>All modern browsers have developer tools so you can examine elements to see what CSS is being applied, and in a surprising departure from the norm, the best one in my opinion is the one available in IE when you press F12.</p>
<p>For example, in this screenshot you can see that the colour being applied here is the one from the class ul.dhtml#nav not the one specified for body or the one specified for body.mainframe.</p>
<img src="/img/ie-dev-tools.png" />
<h2 id="inheritance">Inheritance</h2>
<p>I could not do a better job of writing up inheritance than <a href="http://www.maxdesign.com.au/divs/css-inheritance/" target="_blank">this</a> company presentation. I really recommend you go and click through it. It starts very simple and you may think it's a waste of your time but if you are not 100% sure how inheritance in CSS works then this will really help.</p>
<p>A brief summary though if you really can't be bothered: Elements inherit styles from the elements within which they are nested. So in the following html, if you apply a style of <span class="pretty-code-sample">font-weight: bold</span> to p, then the text inside the span will also be bold.</p>
<p class="pretty-code-sample">
‹body›<br/>
‹p›I'm bold. ‹span›And so am I!‹/span›‹/p›<br/>
‹/body›
</p>
<p><a href="#evaluation-of-css">Read on</a> for how to write it efficiently!</p>
A Voyage of Discovery – Upgrading Firefox Part 22011-07-27T00:00:00+00:00https://www.annashipman.co.uk/jfdi/upgrading-firefox-part-2.html<p>Right, it turned out my next step was not a chat with Joe Halliwell but
instead with my good friend and mentor, <a href="https://twitter.com/boffhaku">Dolan O'Toole</a>, and what I thought might be a voyage of discovery into how to make my installation of firefox run like a normal program turned out to be just a short trip of discovery to finding out that I hadn't actually been looking in the right places.</p>
<p>After helpfully pointing out that I "sound like a linux newbie :)" (yes, yes I am), Dolan explained where I'd gone wrong: "You followed the instuctions as they are. The only thing you weren't aware of is that most of the time, someone would have done a build of a program for ubuntu and put it in an apt repository, usually on launchpad. I usually search for something like "firefox 5 ubuntu 10.10" if I need to install something before I resort to manual installations."</p>
<p>Good tip, and here it is:</p>
<p><a href="http://www.webupd8.org/2011/06/firefox-5-lands-in-firefox-stable-ppa.html">http://www.webupd8.org/2011/06/firefox-5-lands-in-firefox-stable-ppa.html</a></p>
<p>So – I have learned that it's not quite as straightforward as finding the instructions and then following them – you also have to know stuff. That's OK though. And at least I have firefox the way I want it now. I'm sure there'll be another opportunity to get my hands dirty...</p>
Upgrading Firefox – part 1...2011-07-20T00:00:00+00:00https://www.annashipman.co.uk/jfdi/upgrading-firefox-part-1.html<p>So. Not content with the Ubuntu version of Firefox which is at 3.6.18, I decided that I wanted Firefox 5.0. Heck, I want 5.0.1.</p>
<p>So I followed <a href="https://help.ubuntu.com/community/FirefoxNewVersion/MozillaBuilds" target="_blank">these</a> instructions.</p>
<p>To no avail. I followed them to the letter (or so I thought), but when I finished up by running <span class="code-sample">firefox</span>, to my joy, Firefox popped up, and to my moments later horror, it was still 3.6.18.</p>
<p>Why??</p>
<p>Well, lots of reasons why, as I discovered when I unistalled Firefox 3.6.18.</p>
<p>First of all, when it said "Run the following shell script" I just typed it into the console window. That did not work. Don't sigh, I'm new to this!</p>
<p>So I created the shell script and saved it. Following, incidentally, instructions <a href="http://www.go2linux.org/starting-with-shell-script" target="_blank">here</a> – the bit I was missing was the permissions: <span class="code-sample">chmod +x myfirstscript.sh</span>
<p>[NB. I couldn't agree more with the grateful commenter on that link: "Everybody just writes "run the shell script" but a complete beginner doesn't know, that shell scripts have to be executable and are started with ./"]</p>
<p>OK I've run the script woo hoo! "The firefox command in your ~/bin directory will now run Firefox with the mozilla-build profile." I have the file, the file contains another script, that script should allow me to start firefox, right? </p>
<p>Wrong.</p>
<p><span class="code-sample">bash: /usr/bin/firefox: No such file or directory</span></p>
<p>Well that, presumably is why running the firefox command before started up the old firefox. (So – hey – I could have just run the script in the command line like I wanted!)</p>
<p>So, I decided to roll back a step and delete the new <span class="code-sample">~/bin</span> directory – I didn't like it anyway. I discovered I could actually run Firefox, from the firefox directory, by running this command: <span class="code-sample">exec "/home/anna/firefox/firefox" -P mozilla-build "\$@" </span>. So I wrote a shellscript to do that.</p>
<p>So now I can run firefox. It's 5.0.1. But I have to run it from the command line. Closing Firefox doesn't close the terminal window, but that terminal window is taken up with running firefox (maybe there's a way round that like with gedit &).</p>
<p>A few other things. The firefox running remembers passwords. It remembers what I've set my home page to. But it doesn't start up on the home page even though I've asked it to. Agreeing to restart it does so. Behind the scenes, there are a couple of errors: </p>
<p class="code-sample">(firefox-bin:1923): Gtk-WARNING **: Loading IM context type 'ibus' failed
(firefox-bin:1923): Gtk-WARNING **: /usr/lib/gtk-2.0/2.10.0/immodules/im-ibus.so: wrong ELF class: ELFCLASS64
</p>
<p>Ideally I want this firefox to run normally, i.e. be available on the menu, have an icon in the top panel, not to rely on a command window. I suspect my next port of call is to discuss this with Mr Joseph Halliwell. I hope there will be a part 2 where I turn out to have learned loads about this area!</p>
What I have learned about folder structure today2011-04-22T00:00:00+00:00https://www.annashipman.co.uk/jfdi/what-i-have-learned-about-folder-structure-today.html<p><a href="http://packages.python.org/distribute/easy_install.html" target="_blank">Easy_install</a> (as it comes) doesn't work for Debian-based systems.</p>
<p>The default behaviour is to install things to <span class="code-sample">/usr/local/</span> – this is the Gnu default – whereas the synaptic package manager installs things to <span class="code-sample">/usr/</span>. I imagine there may be a way to reconfigure the default behaviour, but I didn't get that far.</p>
<p>This is how I found out this rather useful piece of information. I had installed easy_install, and then used it to download all the dependencies I needed to get started with this python/Django project I was going to work on. And then – nada.</p>
<p>For about three hours, no matter what I did, I had this message: </p>
<p class="code-sample">File "/usr/local/lib/python2.6/dist-packages/Django-1.3-py2.6.egg/django/db/backends/mysql/base.py", line 28, in <module>
from django.db import utils
ImportError: cannot import name utils
</p>
<p>My friend who was with me, trying to get me set up to start work on the project, is a Django expert, but he uses Windows. Hey, so do I! What am I even doing using linux? (See sidebar for answer...)</p>
<p>We tried a lot of things. I won't go into them all here, mainly because I can't remember many of them. We tried a lot of things that people on the internet had suggested. As a last resort, we even tried to RTFM. To no avail.</p>
<p>After we'd given up and my friend had left, I went back to it to try and figure out what was going on. I even tried to resort to the beginner standard of adding log messages, but all the files I wanted to edit were read-only. Finally, <a href="http://docs.djangoproject.com/en/dev/topics/install/?from=olddocs" target="_blank">this page</a> offered me a glimmer of a clue.</p>
<p>Specifically, the question "Where are my site-packages stored?" and the answer:</p>
<p><span class="code-sample">python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"</span></p>
<p>Which for me, returns <span class="code-sample">/usr/lib/python2.6/dist-packages</span></p>
<p>Hang on though, the error is in <span class="code-sample">/usr/local/lib</span> . . . eh?</p>
<p>A visit to the python2.6 folder in <span class="code-sample">/usr/local</span> confirmed that all of the stuff I needed was in there. And a call to my incredibly helpful and talented friend <a href="http://www.joehalliwell.com/" target="_blank">Joe Halliwell</a> confirmed my growing suspicion that this was Not Correct. He explained to me the Gnu/Debian folder structure differences.</p>
<p id="no-sorl-thumbnail">So, I backtracked. I installed everything I needed (except sorl-thumbnail, which wasn't there. However, all I need to do is find a ppa. There is probably also another way to install it correctly. By the time this is all done, I may even know what it is.)</p>
<p>I ran through a few errors on the way, but all of the <span class="code-sample">ImportError: No module named django_extensions</span> variety, and solved one by one, satisfyingly, by installing what I needed in the correct way.</p>
<p>Finally, several hours after my friend came round to work on this, I now have a new error:</p>
<p class="code-sample">File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 170, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'anna'@'localhost' (using password: NO)")
</p>
<p>It's progress.</p>