In my previous article about principal engineers, I have laid out some basic thinking principles behind having more principal engineers. I have also stated and explained my disagreement to use the word architect in software engineering world. Today I would like to explore more into what principal engineers are.
“Every technology is both a burden and a blessing; not either-or, but this-and-that.”
Neil Postman, Technopoly: The Surrender of Culture to Technology
Recently I had come into contact of a phrase grumpy old men in engineering context. My initial reaction have been … what? In today’s world of fast moving, proactive and positive thinking, why would one need any critical voice trying to tone us down? We have just pumped ourselves with positive energy to deliver that product, solve that problem, earn more money, move forward with our lives.
The book of Neil Postman, which I have quoted above explains well, why this one-sided thinking might be joyously misleading. It also deals well with why you need to have grumpy, experienced, principal engineers in the company. People with invaluable knowledge, who are able to challenge each move forward, based on what they been through so far.
The 1996 book by Robert Dilts, titled: Visionary Leadership Skills: Skills and Tools for Creative Leadership puts a lot of focus on the critical side of the creative process. It moves away from just thinking how to move forward, talking to people who like your ideas, but puts focus on running your ideas by people who are known for seeing what you might have missed. In that sense commoditizing negativity.
The Lean Startup: How Constant Innovation Creates Radically Successful Businesses deals with it from a similar perspective. It teaches you to pivot, as soon as you have learned something, even if you don’t like it. The path to the eventual success here, is based on knowing and measuring, then listening to change. It never tells you, all your ideas will be great. Without the critical and yes sometimes necessarily negative assessment, you can’t get there. A successful startup will pivot a number of times before it succeeds. The ability to pivot and respond to learnings and knowledge is also understood as one of the key factors to success.
There are also multiple studies by Robert G. Cooper, which put a lot of focus on activities that are executed before you start working on a new product. The two key ones are: planning before developing, technological synergy and quality.
Technological synergy is a measure of how far from their current technology a firm must stray to build the product. The further away they stray, the less likely the project will be successful. Again the spread is broad: those in the top 20% had an 80% chance of success, those in the bottom 20% had a 29% chance. Beyond synergy itself, although clearly influenced by it, is the quality of technological activities – still a major challenge for many businesses – and one which significantly impacts success.
Planning before developing deals with: how thoroughly and clearly have you understood the product before we begin development? I’ve often seen companies leap straight to development before a proper understanding of the product has been created. By deferring execution in favour of better definition (of the market and technical and business characteristics of the product) we significantly increase the odds of success.
Cooper emphasises that this definition “must be based on solid evidence, not speculation” (p12): “In too many projects, we witnessed a product idea that moved directly into development with very little in the way of homework to define the product and justify the project. More often than not, the results were negative.”
Cooper, R. and Elko Kleinschmidt. 1990. New Products, Key Factors in Success. Stage-Gate International.
The key question that appears here is: how can you do it with just a bunch of yes-men trying to please the board, markets, themselves? The answer is you can’t. Principal engineers and their experience can help you get there, even if they are always just grumpy old men.
Besides just the allowance for criticism, open challenge and people being able to pull the hand-brake, described in the paragraphs above, let’s explore more what Robert G. Cooper calls technological synergy. Is it yet another catchy phrase, so common for software engineering world? Is it a thing like digital transformation, which became so common nowadays, but it still fails to give you a good enough definition to apply it practically?
The associations of technology synergy, product characteristics, and new product performance are widely spread in the marketing and innovation management literatures. However, little research integrates these associations. This study adopts a meta‐analytic approach to aggregate prior findings across studies published before 2010 to review the relationships between technology synergy, product characteristics, and new product performance. Structural equation analysis reveals that technology synergy has: (a) a positive medium effect on new product performance; (b) a positive and strong impact on product advantage, which then affects new product performance; and (c) an indirect effect on new product performance through product innovativeness and product advantage. These findings suggest that product innovation and advantage are important intermediaries between technology synergy and new product performance—as yet unrevealed in extant literature.
Technology Synergy, Product Characteristics, and New Product Performance: A Meta‐Analytic Review, Kuen‐Hung Tsai, Chi‐Tsun Huang
Based on the paper quoted above, technology synergy is a capability that enables an enterprise to improve product innovation, and in turn gain a competitive advantage. A firm with a greater level of technology synergy possesses more technological resources needed in the development of new products. To cut the long story short – companies with high technological synergy, are using technologies that can be relatively easily understood / have easy access to resources (think COBOL vs java or python), can hire talent easily and are able to operate it without huge overhead. This should also leave the area of rapid innovation open.
IT is most likely proving why cloud platforms are making a huge difference for businesses today. The shift to outsource complexity (as described in my article on quality), especially on the low level stuff, this being servers and operating systems, automatically increases the technological synergy. When you bind all that with inventions like graphql, react or other js-based libs for frontend (which outsource the complexity of handling data aggregation, various browser stacks and different display devices), you are nearing the sweet spot.
What is the role of principal engineers here then?
I have heard multiple opinions on how outsourcing complexity is bad. You don’t understand things you use anymore, performance tuning becomes a problem, etc, etc. Still making things too bespoke for your company (as based on multiple research papers), while it might make some senior engineers happy, is bad for business.
The role of principal engineers here is to go and explain this idea and coach others on the benefits. Mark my words, as I am not saying they should go ahead and suggest architectures or frameworks. These are at best … best practices and nothing else. They should not be a prescribed rule to success. For example I have personally nothing against the AWS direction of Well Architected Framework, still some things can be made better without using the fancy suggestions from AWS. I am particularly not a huge fan of always using security-first, which in many cases will mean performance-second. As an example – does your service never exposed to external world, used by internal applications with non-critical data, need the IAM-level authentication? Security is a matter of assessing and accepting risks, not a prescription making service access slower and more complex through the additional authentication logic.
And here I am, a grumpy old engineer, complaining … 🙂
What else can your principal engineers do? To achieve the sweet spot of high technological synergy, apart of recommending and coaching others (including product and whole business, not just engineers) in best practices, you also need to understand where to make a decision between extend and rebuild.
A lot of services in the world are built just because (literally no other explanation). Technological synergy teaches us – value comes from continuous innovation. Change is inevitable. The risk is though going down the drain of my engineers told me so. The decision whether a service should be extended or rebuilt has many facets. One of them is hands-on experience, which your principal engineers must have. They also need to continuously learn about the existing systems. There is no end to that and even a system you know for over 10 years, can still surprise you. Second is the ability to understand where markets in hiring are going, whether the firm will be able to find the right talent. Again this means being hands on with what’s going on the market. Being up to date with novelties. Third and probably most important is either shifting or understanding development community sentiment towards a specific system. No project will succeed from engineering perspective, if the technology is not liked by the engineering community. Try introducing .Net to a unix-first company, or ask java developers to switch to python. High attrition is expected.
The above helps principal engineers to stop accepting the status-quo of systems being built just-because. Also allows them to help guide the future direction and instill a spirit of innovation. In the end, like change, innovation is inevitable in engineering world. Engineers are never fully happy with what they have built.
To repeat high technological synergy, is using technologies that can be easily understood / have easy access to resources, where one can hire talent easily and is able to operate it without huge overhead. This should also leave the area of rapid innovation open.
Wrapping up the post for today with key highlights:
- each technology you use has pros and cons
- each product requires a lot of critical and sometimes (OMG!) negative thinking to succeed
- there is no silver bullet to solve that, just do-learn-adjust
- you need grumpy old engineers to understand that and prevent all-happy, rainbow and unicorns approach
- technological synergy is very important
- outsourcing complexity and using cloud solutions helps to increase technological synergy
- again, you’ll need your principal engineers to help build higher technological synergy
- python is better than java (who … me? I never said that 🙂 )
To not be one-sided though, below is a great video on the ‘feasibility trap’. Basically if you are too good at critical thinking, it might result in harming your personal career as an engineer. Even if the products you eventually build are superstar due to that.
What should one do then? My thinking is – criticise with a smile? Like in the famous – maybe bad news delivered in a good way sound better? Don’t really know the answer. At the same time, I personally believe there is an equivalent amount of value in frequently saying hold on gang, comparable to going gung-ho and just doing things.
And just maybe … rest of business should spend more time understanding engineers?
Thanks and see ya next time 🙂
Gustaw Fit 