Name: Is strong data infrastructure the ultimate insights tool?
Uploaded: 2024-02-20T13:29:05.384Z
Duration: 373.71

Data Infrastructure V7.mp4

data infrastructure.vtt

English

{{00:00:00}}
In this wild world where
you're always trying to find a

{{00:00:02}}
competitive advantage against your
competition, data is your asset.

{{00:00:16}}
Data infrastructure refers to, I
guess, the systems and platforms that

{{00:00:21}}
you use to, I would say, you know,
ingest, store, transform, manage, and

{{00:00:27}}
serve data, for a number of use cases.

{{00:00:30}}
For analytical use cases, for
reporting, for machine learning, for

{{00:00:33}}
data science, for personalisation,
really anything, any systems that,

{{00:00:38}}
that use data in your organisation,

{{00:00:46}}
It's important for, for, for companies
to invest in data infrastructure,

{{00:00:50}}
because, you know, in this wild world
where you're always trying to find

{{00:00:54}}
a competitive advantage against your
competition, data is your asset, and if

{{00:00:58}}
you're not storing it, monetising it,
taking advantage of it, using it,  you

{{00:01:03}}
know, your competitors are, and it's kind
of your lost opportunity to not invest

{{00:01:08}}
in data infrastructure at this point.

{{00:01:10}}
I would say data
infrastructure is table stakes

{{00:01:19}}
At the center of all data infrastructure.

{{00:01:21}}
You have some sort of platform that
you're storing data that's either

{{00:01:24}}
a data warehouse or a data lake.

{{00:01:26}}
Those are things like,
Snowflake, BigQuery, Redshift,

{{00:01:30}}
Databricks, Azure Synapse.

{{00:01:32}}
And then you need to get data
into the into the data warehouse.

{{00:01:36}}
And you're using the ingestion platform
typically as tools like, I would say,

{{00:01:40}}
Fivetran,  Meltano, Azure Data Factory.

{{00:01:46}}
You know, self built integrations,
and then once the data is in

{{00:01:51}}
the data warehouse, we follow
what's called an ELT approach.

{{00:01:55}}
So extract, load, and then you
transform in the data warehouse.

{{00:02:00}}
And you're going to use a tool potentially
like DBT or Python scripts to transform

{{00:02:04}}
the data into, I describe it as an
analytics optimised data schema.

{{00:02:09}}
Essentially, tables that are easier
for analysis and reporting than

{{00:02:13}}
the raw data that you ingested.

{{00:02:15}}
And then you'll typically have some
sort of BI or visualisation tool.

{{00:02:19}}
That's things like Sigma,
Looker, Tableau,  Power BI.

{{00:02:24}}
And that's kind of the core components.

{{00:02:26}}
There's a lot of other things
like observability, data

{{00:02:28}}
catalogs, orchestration.

{{00:02:29}}
But, but, you know, between integration,
data warehouse, your transformation

{{00:02:33}}
platform, and your visualisation
tool, that's kind of the core

{{00:02:36}}
component of the data infrastructure.

{{00:02:44}}
Potentially, upstream
issues can flow downstream.

{{00:02:47}}
And so as we're getting more and more
connected, Say you are theoretically

{{00:02:51}}
driving personalised email
campaigns off your data warehouse.

{{00:02:54}}
There is risk that if you're
ingesting bad data, you might trigger

{{00:02:58}}
kind of potentially, you know, bad
campaigns to run in production,

{{00:03:01}}
sending emails to the wrong people.

{{00:03:03}}
And so that's why you're typically
going to be running your data

{{00:03:06}}
infrastructure and designing in, I
would say, an deffensive pattern.

{{00:03:10}}
So using data quality tests,  that check
for uniqueness,  counts,  you're just

{{00:03:17}}
differences from, from known thresholds.

{{00:03:19}}
And so having some sort of observability
tools and, and test that, you know,

{{00:03:23}}
if they fail, that it, that it can
kind of stop the process or like

{{00:03:27}}
pull the cord on the assembly line.

{{00:03:29}}
That, that's really important as
you're starting to run more production

{{00:03:32}}
workloads that have, let's say, external
impacts like driving personalisation

{{00:03:36}}
off of your data warehouse.

{{00:03:43}}
Everybody uses AI and ML,
you just don't know it.

{{00:03:46}}
You're often using SaaS tools that
are using your data and is doing

{{00:03:50}}
machine learning and AI behind the
scenes, and, you know, it feels like

{{00:03:53}}
magic or it's intuitive, but that's
actually AI and machine learning.

{{00:03:56}}
I think the question is, should you
internalise some of that yourself?

{{00:04:01}}
And I think it's very company specific.

{{00:04:03}}
I don't think it's an obligation.

{{00:04:05}}
I think it's a journey.

{{00:04:06}}
So you, you invest in the
foundation, focus essentially, I

{{00:04:10}}
like to think about it on climbing
the, the data hierarchy of needs.

{{00:04:14}}
First, you're going to clean
your data, and then you're going

{{00:04:16}}
to potentially have dashboards.

{{00:04:18}}
Then you're going to do analysis and
make decisions, and potentially you can

{{00:04:22}}
get to the point where machine learning
is helping you to recommend and maybe

{{00:04:26}}
automate an action, but I would say like
that's further up the hierarchy of need.

{{00:04:30}}
So I would say feel no obligation to
go straight to AI and machine learning

{{00:04:34}}
because odds are you're probably
already using it in the tools you

{{00:04:36}}
already use and you don't know it.

{{00:04:44}}
We're in an interesting spot with AI.

{{00:04:46}}
I think we're in the spot where
we all universally agree that

{{00:04:48}}
AI will change everything.

{{00:04:51}}
But we don't know how.

{{00:04:52}}
And so we're at this point where I think
AI is very similar to autonomous vehicles.

{{00:04:57}}
We all realise that it's going
to be harder than we thought to

{{00:05:01}}
get to fully autonomous vehicles.

{{00:05:03}}
And so, you know, instead of
waiting till we have the perfect

{{00:05:06}}
car that can drive at any street.

{{00:05:08}}
We need to prepare the
streets for the cars.

{{00:05:10}}
And so that's exactly what you
do with data infrastructure.

{{00:05:13}}
I would say by organising your
data, organising the metadata

{{00:05:17}}
structuring in such a way that
it's easier for AI, generative AI

{{00:05:21}}
to parse it and answer questions.

{{00:05:24}}
I think that's the step in the
direction of preparing for AI.

{{00:05:27}}
We can't wait for the, the,  cars
to be prepared for the roads.

{{00:05:31}}
We have to prepare the roads for the cars.

{{00:05:39}}
My favourite you know, component
of the data stack, I would say,

{{00:05:42}}
is the transformation layer.

{{00:05:43}}
I mean, it's, it's great to grab the,
pull the data into your data warehouse.

{{00:05:47}}
It's great to have a data
warehouse that stores the data.

{{00:05:49}}
And it's great to have a BI tool.

{{00:05:51}}
But I think, for me, the
transformation layer is, is where

{{00:05:54}}
you're codifying business logic.

{{00:05:56}}
You're, you're, kind of developing
and massaging the data in a way that

{{00:06:01}}
makes it easier for people to use.

{{00:06:02}}
I mean that, I feel like the
transformation layer is where

{{00:06:05}}
it all comes together and
that's where the magic happens.

In this wild world where
you're always trying to find a

competitive advantage against your
competition, data is your asset.

Data infrastructure refers to, I
guess, the systems and platforms that

you use to, I would say, you know,
ingest, store, transform, manage, and

For analytical use cases, for
reporting, for machine learning, for

data science, for personalisation,
really anything, any systems that,

It's important for, for, for companies
to invest in data infrastructure,

because, you know, in this wild world
where you're always trying to find

a competitive advantage against your
competition, data is your asset, and if

you're not storing it, monetising it,
taking advantage of it, using it,  you

know, your competitors are, and it's kind
of your lost opportunity to not invest

I would say data
infrastructure is table stakes

At the center of all data infrastructure.

You have some sort of platform that
you're storing data that's either

Those are things like,
Snowflake, BigQuery, Redshift,

And then you need to get data
into the into the data warehouse.

And you're using the ingestion platform
typically as tools like, I would say,

You know, self built integrations,
and then once the data is in

the data warehouse, we follow
what's called an ELT approach.

So extract, load, and then you
transform in the data warehouse.

And you're going to use a tool potentially
like DBT or Python scripts to transform

the data into, I describe it as an
analytics optimised data schema.

Essentially, tables that are easier
for analysis and reporting than

And then you'll typically have some
sort of BI or visualisation tool.

That's things like Sigma,
Looker, Tableau,  Power BI.

There's a lot of other things
like observability, data

But, but, you know, between integration,
data warehouse, your transformation

platform, and your visualisation
tool, that's kind of the core

Potentially, upstream
issues can flow downstream.

And so as we're getting more and more
connected, Say you are theoretically

driving personalised email
campaigns off your data warehouse.

There is risk that if you're
ingesting bad data, you might trigger

kind of potentially, you know, bad
campaigns to run in production,

And so that's why you're typically
going to be running your data

infrastructure and designing in, I
would say, an deffensive pattern.

So using data quality tests,  that check
for uniqueness,  counts,  you're just

And so having some sort of observability
tools and, and test that, you know,

if they fail, that it, that it can
kind of stop the process or like

That, that's really important as
you're starting to run more production

workloads that have, let's say, external
impacts like driving personalisation

Everybody uses AI and ML,
you just don't know it.

You're often using SaaS tools that
are using your data and is doing

machine learning and AI behind the
scenes, and, you know, it feels like

magic or it's intuitive, but that's
actually AI and machine learning.

I think the question is, should you
internalise some of that yourself?

So you, you invest in the
foundation, focus essentially, I

like to think about it on climbing
the, the data hierarchy of needs.

First, you're going to clean
your data, and then you're going

Then you're going to do analysis and
make decisions, and potentially you can

get to the point where machine learning
is helping you to recommend and maybe

automate an action, but I would say like
that's further up the hierarchy of need.

So I would say feel no obligation to
go straight to AI and machine learning

because odds are you're probably
already using it in the tools you

I think we're in the spot where
we all universally agree that

And so we're at this point where I think
AI is very similar to autonomous vehicles.

We all realise that it's going
to be harder than we thought to

And so, you know, instead of
waiting till we have the perfect

We need to prepare the
streets for the cars.

And so that's exactly what you
do with data infrastructure.

I would say by organising your
data, organising the metadata

structuring in such a way that
it's easier for AI, generative AI

I think that's the step in the
direction of preparing for AI.

We can't wait for the, the,  cars
to be prepared for the roads.

We have to prepare the roads for the cars.

My favourite you know, component
of the data stack, I would say,

I mean, it's, it's great to grab the,
pull the data into your data warehouse.

It's great to have a data
warehouse that stores the data.

But I think, for me, the
transformation layer is, is where

You're, you're, kind of developing
and massaging the data in a way that

I mean that, I feel like the
transformation layer is where

it all comes together and
that's where the magic happens.

Expert Insight

Data infrastructure is the digital scaffolding that houses a multitude of internal systems for data collation and analysis. Making these structures sturdy and reliable can be the difference between having a competitive advantage or falling short. 
In this video, Scott Breitenother, Founder & CEO of Brooklyn Data Co, investigates the elements of data infrastructure and how companies can harness them for strong business insights.

Data & Analytics

Data Collection

Data analytics

Analysis

AI-ML

Journey Analytics

Data Warehouse

Customer Data Platform

Machine Learning and AI

 Insights

Data

thumbnail_Scott Breitenother.jpg

xsmall_Scott Breitenother.jpg

Scott Breitenother.jpg

Scott Breitenother

Founder and CEO

Breitenother explores the key components of any strong data infrastructure system.

Core components of data infrastructure

Breitenother contends with some of the challenges of a connected data infrastructure.


Challenges of a connected data infrastructure

A hierarchy of needs is proposed by Breitenother when considering AI and ML data infrastructure. 