Data Foundations For Artificial Intelligence - How Connecting Silos Will Be Key

At a recent conference, a most enlightened debate - about "what is an ideal data foundation for AI" - delivered some incredibly exciting and remarkably unorthodox revelations. One the most radical ideas shared - and debated - was the creative approach of "connecting silos".

Disliked, detested and perhaps even despised by some, but could silos find new meaning and - indeed - make an important contribution towards enabling robust data foundations for Artificial Intelligence? And could a most pertinent question to be asked be: rather than being torn down, could we possibly see a resurgence of silos and a situation where silos are actually actively encouraged?

What are silos?

Traditionally, a silo is a system, a process, a data set, a function, a department - or similar collective - that operates in isolation from others. However, over the years, the idea of silos has evolved: it has become perceived to be the embodiment of the friction and the inertia that hamper organisations.

What is so "wrong" about them (and how have they acquired such a bad name)?

• Silos are seldom adaptable and are typically inflexible: most silos do not cope well with change and struggle to transform, as use cases and demands vary, evolve or become more complex.
• Silos typically struggle to connect to anything or anyone outside their boundaries: a characteristic that renders a lot of silo environments, muddled, disjointed and somewhat fragmented.

What is so "right" about them (and how have they become key for AI foundations)?

• An extremely interesting attribute - of a typical silo - is its absolute focus on its purposed use case(s) or function(s) and nothing else. This can be an incredibly beneficial attribute for scope management, minimum viable product engineering and for delivering tangible value iteratively.
• Given their singleminded focus and given their leaning towards specialisation, silos are typically particularly efficient for their intended or associated purpose(s). This attribute can lead to demonstrable improvements in the quality of delivered outcomes.

"How to" bring it all together (connecting silos)?

So, what if we could have the best of both worlds: focused and efficient, yet, adaptable and connected?

The overarching approach to "connecting silos" is not only underpinned by "connecting" but is actually empowered by building in "silos" as well. How?

• Identify (or create) boundary relationships (or connections) between silos;
• Establish and enable access gateways - from one silo to the others;
• Promote standardisation of language(s) and methods of communication across silos

What key areas would (connecting silos) help accelerate?

• People: a move from the heritage, traditional and predominantly “functional silo" model towards a more empowering and more scalable "autonomous silo" approach. This would reduce the need to have to jump over one fence - and then the other, and then the other - to be enable to assemble the capabilities required, to get a job done, and in order to enable more effective interactions.
• Design: with a focus on adaptability and scalability; creatively designing - and building - one silo at a time, but, with the clarity of vision to determine how each silo will be able to connect to all others.
• Lifecycle: the key would be to deliver one silo; and then another, and then another - and to deliver into "production" in small but frequent increments.

What not to do (pitfalls and gotchas)?

• Rigid organisation of people around projects can be quite a compelling proposition (as it could appear to enable much firmer resource management control than alternative approaches) but this can have the devastating effect of limiting flexibility. Another significant consideration is that project teams ultimately disband - and as such become detached from any accountability regarding the quality, resilience or operational effectiveness of their deliverables. Instead, organising people around the actual service(s) that they provide (and "own") would help establish much more sustainable data foundations for AI.
• Limiting toolsets that can be used for consumption is a key reason why silos, traditionally, emerged in the first instance; the lack of flexibility or capability leaving consumers with no choice but to move data towards their required tools and capabilities. Instead, an intentional democratisation of tools and data access (with no more control than appropriate) is more likely to deliver the ideal basis for AI.
• Assuming all consumers have the same needs or capabilities (e.g., attempting to have everyone "self serve" to similar levels of sophistication or restricting all consumers to a standardised finished product) is a key challenge that often leads to strategic failure. Instead, considering the diverse range of consumer needs - and capabilities - and adapting data services, as appropriate for each identified cohort or category, is much more likely to deliver success.

In summary

So, when we think about silos - rather than considering that the only way out might be to pull them right down to the ground - we can now consider that a viable alternative would be connecting them. And when we think about "connecting silos", we can feel more empowered to approach such opportunities by considering appropriate models for organising our people, our designs, and our lifecycle. And as we contemplate "what to do" - differently - for the future, it is perhaps just as important not to forget "what not to do".

Thank you